Piping streams together in Node.js

Introduction

Piping streams lets you connect data sources and destinations easily. It helps move data step-by-step without loading everything in memory.

Reading a file and writing its content to another file

Downloading data from the internet and saving it directly to disk

Processing large data in chunks, like compressing or encrypting on the fly

Connecting multiple data transformations in a chain without manual buffering

Syntax

Node.js

readableStream.pipe(writableStream);

The pipe() method connects a readable stream to a writable stream.

It automatically manages data flow and backpressure.

Examples

This copies the content of input.txt to output.txt using streams.

Node.js

const fs = require('node:fs');
const readable = fs.createReadStream('input.txt');
const writable = fs.createWriteStream('output.txt');
readable.pipe(writable);

This reads a file, compresses it with gzip, and writes the compressed data to a new file.

Node.js

const zlib = require('node:zlib');
const fs = require('node:fs');
const readable = fs.createReadStream('input.txt');
const gzip = zlib.createGzip();
const writable = fs.createWriteStream('input.txt.gz');
readable.pipe(gzip).pipe(writable);

Sample Program

This program reads example.txt, compresses it using gzip, and saves it as example.txt.gz. When done, it logs a success message.

Node.js

import fs from 'node:fs';
import zlib from 'node:zlib';

const readable = fs.createReadStream('example.txt');
const gzip = zlib.createGzip();
const writable = fs.createWriteStream('example.txt.gz');

readable.pipe(gzip).pipe(writable);

writable.on('finish', () => {
  console.log('File compressed successfully');
});

Important Notes

Always handle errors on streams to avoid crashes.

Use the 'finish' event on writable streams to know when writing is done.

Piping helps keep memory usage low by processing data in chunks.

Summary

Piping connects readable and writable streams simply.

It is useful for copying, transforming, or compressing data streams.

It manages data flow automatically, making code cleaner and efficient.

Practice

1. What is the main purpose of piping streams together in Node.js?

easy

A. To pause and resume streams manually

B. To convert streams into arrays for easier processing

C. To connect a readable stream directly to a writable stream for automatic data flow

D. To create new streams from scratch

Solution

1. Step 1: Understand what piping does

   Piping connects a readable stream to a writable stream so data flows automatically without manual intervention.

2. Step 2: Compare options

   Only To connect a readable stream directly to a writable stream for automatic data flow describes this automatic connection and data flow. Other options describe unrelated stream operations.

3. Final Answer:

   To connect a readable stream directly to a writable stream for automatic data flow -> Option C

4. Quick Check:

   Piping = automatic stream connection [OK]

Hint: Piping means connecting streams for automatic data transfer [OK]

Common Mistakes:

• Thinking piping converts streams to arrays
• Confusing piping with manual pause/resume
• Assuming piping creates new streams

2. Which of the following is the correct syntax to pipe a readable stream readStream into a writable stream writeStream?

easy

A. readStream.pipe(writeStream);

B. writeStream.pipe(readStream);

C. pipe(readStream, writeStream);

D. readStream.write(writeStream);

Solution

1. Step 1: Recall pipe method usage

   The pipe() method is called on a readable stream and takes a writable stream as argument.

2. Step 2: Check each option

   readStream.pipe(writeStream); matches the correct syntax. writeStream.pipe(readStream); reverses streams, C uses a non-existent function, D misuses write method.

3. Final Answer:

   readStream.pipe(writeStream); -> Option A

4. Quick Check:

   Readable.pipe(Writable) = correct syntax [OK]

Hint: Remember: readableStream.pipe(writableStream) [OK]

Common Mistakes:

• Reversing the order of streams in pipe
• Using pipe as a standalone function
• Calling write instead of pipe

3. Consider this code snippet:

const fs = require('fs');
const readStream = fs.createReadStream('input.txt');
const writeStream = fs.createWriteStream('output.txt');
readStream.pipe(writeStream);
writeStream.on('finish', () => console.log('Done'));

What will be printed when the piping finishes?

medium

A. No output

B. Error

C. undefined

D. Done

Solution

1. Step 1: Understand the pipe and event

   The readable stream pipes data to the writable stream. When writing finishes, the 'finish' event triggers.

2. Step 2: Check the event handler

   The code listens for 'finish' on writeStream and logs 'Done' when triggered.

3. Final Answer:

   Done -> Option D

4. Quick Check:

   'finish' event logs 'Done' [OK]

Hint: Listen to 'finish' event on writable stream for completion [OK]

Common Mistakes:

• Expecting 'end' event on writable stream
• Not handling asynchronous event
• Confusing 'finish' with 'close'

4. What is wrong with this code snippet?

const fs = require('fs');
const readStream = fs.createReadStream('file.txt');
const writeStream = fs.createWriteStream('copy.txt');
writeStream.pipe(readStream);

medium

A. The file paths are incorrect

B. The pipe method is called on the writable stream instead of the readable stream

C. Missing error handling on streams

D. Streams cannot be piped in Node.js

Solution

1. Step 1: Identify pipe usage

   The pipe method must be called on a readable stream, passing a writable stream as argument.

2. Step 2: Analyze the code

   The code calls pipe on writeStream (writable), which is incorrect and will cause an error.

3. Final Answer:

   The pipe method is called on the writable stream instead of the readable stream -> Option B

4. Quick Check:

   Readable.pipe(Writable) only [OK]

Hint: Pipe is always called on readable stream [OK]

Common Mistakes:

• Calling pipe on writable stream
• Ignoring error handling (not main error here)
• Assuming streams can't be piped

5. You want to read data from input.txt, compress it using zlib's gzip, and write the compressed data to output.gz. Which code snippet correctly pipes these streams together?

hard

A. const fs = require('fs');
const zlib = require('zlib');
const readStream = fs.createReadStream('input.txt');
const gzip = zlib.createGzip();
const writeStream = fs.createWriteStream('output.gz');
readStream.pipe(gzip).pipe(writeStream);

B. const fs = require('fs');
const zlib = require('zlib');
const readStream = fs.createReadStream('input.txt');
const gzip = zlib.createGzip();
const writeStream = fs.createWriteStream('output.gz');
gzip.pipe(readStream).pipe(writeStream);

C. const fs = require('fs');
const zlib = require('zlib');
const readStream = fs.createReadStream('input.txt');
const gzip = zlib.createGzip();
const writeStream = fs.createWriteStream('output.gz');
writeStream.pipe(gzip).pipe(readStream);

D. const fs = require('fs');
const zlib = require('zlib');
const readStream = fs.createReadStream('input.txt');
const gzip = zlib.createGzip();
const writeStream = fs.createWriteStream('output.gz');
readStream.pipe(writeStream).pipe(gzip);

Solution

1. Step 1: Understand the stream flow

   Data flows from readable (input.txt) to transform (gzip) to writable (output.gz).

2. Step 2: Check pipe chaining

   const fs = require('fs'); const zlib = require('zlib'); const readStream = fs.createReadStream('input.txt'); const gzip = zlib.createGzip(); const writeStream = fs.createWriteStream('output.gz'); readStream.pipe(gzip).pipe(writeStream); correctly pipes readStream into gzip, then gzip into writeStream. Other options reverse or misuse pipe order.

3. Final Answer:

   readStream.pipe(gzip).pipe(writeStream); -> Option A

4. Quick Check:

   Readable -> Transform -> Writable pipe chain [OK]

Hint: Chain pipes: readable.pipe(transform).pipe(writable) [OK]

Common Mistakes:

• Reversing pipe order
• Calling pipe on writable stream
• Not chaining transform stream correctly