Piping streams together in Node.js

Introduction

Piping streams lets you connect data sources and destinations easily. It helps move data step-by-step without loading everything in memory.

Reading a file and writing its content to another file

Downloading data from the internet and saving it directly to disk

Processing large data in chunks, like compressing or encrypting on the fly

Connecting multiple data transformations in a chain without manual buffering

Syntax

Node.js

readableStream.pipe(writableStream);

The pipe() method connects a readable stream to a writable stream.

It automatically manages data flow and backpressure.

Examples

This copies the content of input.txt to output.txt using streams.

Node.js

const fs = require('node:fs');
const readable = fs.createReadStream('input.txt');
const writable = fs.createWriteStream('output.txt');
readable.pipe(writable);

This reads a file, compresses it with gzip, and writes the compressed data to a new file.

Node.js

const zlib = require('node:zlib');
const fs = require('node:fs');
const readable = fs.createReadStream('input.txt');
const gzip = zlib.createGzip();
const writable = fs.createWriteStream('input.txt.gz');
readable.pipe(gzip).pipe(writable);

Sample Program

This program reads example.txt, compresses it using gzip, and saves it as example.txt.gz. When done, it logs a success message.

Node.js

import fs from 'node:fs';
import zlib from 'node:zlib';

const readable = fs.createReadStream('example.txt');
const gzip = zlib.createGzip();
const writable = fs.createWriteStream('example.txt.gz');

readable.pipe(gzip).pipe(writable);

writable.on('finish', () => {
  console.log('File compressed successfully');
});

Important Notes

Always handle errors on streams to avoid crashes.

Use the 'finish' event on writable streams to know when writing is done.

Piping helps keep memory usage low by processing data in chunks.

Summary

Piping connects readable and writable streams simply.

It is useful for copying, transforming, or compressing data streams.

It manages data flow automatically, making code cleaner and efficient.