Piping streams together in Node.js - Step-by-Step Execution

Concept Flow - Piping streams together

Create Readable Stream

↓

Create Writable Stream

↓

Pipe Readable -> Writable

↓

Data flows chunk by chunk

↓

Writable receives data and writes

↓

End event triggers when done

↓

Streams close

This flow shows how data moves from a readable stream into a writable stream using pipe, chunk by chunk, until all data is transferred and streams close.

Execution Sample

Node.js

const fs = require('fs');
const readable = fs.createReadStream('input.txt');
const writable = fs.createWriteStream('output.txt');
readable.pipe(writable);

This code reads data from 'input.txt' and writes it into 'output.txt' by piping the streams together.

Execution Table

Step	Action	Readable Stream State	Writable Stream State	Data Chunk	Effect
1	Create readable stream	open, ready to read	not created	none	Readable stream ready to emit data
2	Create writable stream	open, ready to read	open, ready to write	none	Writable stream ready to receive data
3	Call readable.pipe(writable)	flowing	waiting for data	none	Pipe connects streams
4	Readable emits first chunk	flowing	writing	chunk1	Writable writes chunk1
5	Readable emits second chunk	flowing	writing	chunk2	Writable writes chunk2
6	Readable emits last chunk	flowing	writing	chunk3	Writable writes chunk3
7	Readable emits 'end' event	ended	waiting for finish	none	Writable finishes writing
8	Writable emits 'finish' event	ended	finished	none	Streams closed
9	Process ends	closed	closed	none	All data transferred, streams closed

💡 Readable stream ends and writable stream finishes, so piping stops.

Variable Tracker

Variable	Start	After Step 3	After Step 4	After Step 7	Final
readable._readableState.flowing	null	true	true	true	true
writable.writableEnded	false	false	false	true	true
data chunks passed	0	0	1	3	3

Key Moments - 3 Insights

Why does the readable stream switch to flowing mode after pipe is called?

Calling pipe() automatically switches the readable stream to flowing mode to start sending data chunks to the writable stream, as shown in execution_table step 3.

What happens when the readable stream emits the 'end' event?

When 'end' is emitted (step 7), it signals no more data will come, so the writable stream finishes writing and emits 'finish' (step 8), ending the pipe.

Can writable stream receive data before pipe() is called?

No, writable stream waits for data until pipe() connects it to the readable stream, as seen in step 2 vs step 3 in execution_table.

Visual Quiz - 3 Questions

Look at the execution table, what is the readable stream's flowing state right after pipe() is called?

A false

B null

C true

D undefined

💡 Hint

Check the 'Readable Stream State' column at step 3 in the execution_table.

At which step does the writable stream finish writing all data?

A Step 7

B Step 8

C Step 5

D Step 9

💡 Hint

Look for the 'Writable emits finish event' in the execution_table.

If the readable stream never emits 'end', what happens to the writable stream?

A It waits indefinitely for more data

B It closes automatically

C It finishes immediately

D It throws an error

💡 Hint

Refer to the 'end' event role explained in key_moments and execution_table step 7.

Concept Snapshot

Piping streams connects a readable stream to a writable stream.
Use readable.pipe(writable) to send data chunk by chunk.
Pipe switches readable to flowing mode automatically.
Writable writes data as it arrives.
Streams close when readable ends and writable finishes.
This simplifies data transfer between streams.

Full Transcript

Piping streams together in Node.js means connecting a readable stream to a writable stream so data flows automatically from one to the other. First, you create a readable stream and a writable stream. Then you call readable.pipe(writable). This switches the readable stream into flowing mode, so it starts emitting data chunks. Each chunk is passed to the writable stream, which writes it out. When the readable stream has no more data, it emits an 'end' event. This tells the writable stream to finish writing and close. The process ends when both streams close. This method makes transferring data simple and efficient without manually handling each chunk.

Practice

1. What is the main purpose of piping streams together in Node.js?

easy

A. To pause and resume streams manually

B. To convert streams into arrays for easier processing

C. To connect a readable stream directly to a writable stream for automatic data flow

D. To create new streams from scratch

Solution

1. Step 1: Understand what piping does

   Piping connects a readable stream to a writable stream so data flows automatically without manual intervention.

2. Step 2: Compare options

   Only To connect a readable stream directly to a writable stream for automatic data flow describes this automatic connection and data flow. Other options describe unrelated stream operations.

3. Final Answer:

   To connect a readable stream directly to a writable stream for automatic data flow -> Option C

4. Quick Check:

   Piping = automatic stream connection [OK]

Hint: Piping means connecting streams for automatic data transfer [OK]

Common Mistakes:

• Thinking piping converts streams to arrays
• Confusing piping with manual pause/resume
• Assuming piping creates new streams

2. Which of the following is the correct syntax to pipe a readable stream readStream into a writable stream writeStream?

easy

A. readStream.pipe(writeStream);

B. writeStream.pipe(readStream);

C. pipe(readStream, writeStream);

D. readStream.write(writeStream);

Solution

1. Step 1: Recall pipe method usage

   The pipe() method is called on a readable stream and takes a writable stream as argument.

2. Step 2: Check each option

   readStream.pipe(writeStream); matches the correct syntax. writeStream.pipe(readStream); reverses streams, C uses a non-existent function, D misuses write method.

3. Final Answer:

   readStream.pipe(writeStream); -> Option A

4. Quick Check:

   Readable.pipe(Writable) = correct syntax [OK]

Hint: Remember: readableStream.pipe(writableStream) [OK]

Common Mistakes:

• Reversing the order of streams in pipe
• Using pipe as a standalone function
• Calling write instead of pipe

3. Consider this code snippet:

const fs = require('fs');
const readStream = fs.createReadStream('input.txt');
const writeStream = fs.createWriteStream('output.txt');
readStream.pipe(writeStream);
writeStream.on('finish', () => console.log('Done'));

What will be printed when the piping finishes?

medium

A. No output

B. Error

C. undefined

D. Done

Solution

1. Step 1: Understand the pipe and event

   The readable stream pipes data to the writable stream. When writing finishes, the 'finish' event triggers.

2. Step 2: Check the event handler

   The code listens for 'finish' on writeStream and logs 'Done' when triggered.

3. Final Answer:

   Done -> Option D

4. Quick Check:

   'finish' event logs 'Done' [OK]

Hint: Listen to 'finish' event on writable stream for completion [OK]

Common Mistakes:

• Expecting 'end' event on writable stream
• Not handling asynchronous event
• Confusing 'finish' with 'close'

4. What is wrong with this code snippet?

const fs = require('fs');
const readStream = fs.createReadStream('file.txt');
const writeStream = fs.createWriteStream('copy.txt');
writeStream.pipe(readStream);

medium

A. The file paths are incorrect

B. The pipe method is called on the writable stream instead of the readable stream

C. Missing error handling on streams

D. Streams cannot be piped in Node.js

Solution

1. Step 1: Identify pipe usage

   The pipe method must be called on a readable stream, passing a writable stream as argument.

2. Step 2: Analyze the code

   The code calls pipe on writeStream (writable), which is incorrect and will cause an error.

3. Final Answer:

   The pipe method is called on the writable stream instead of the readable stream -> Option B

4. Quick Check:

   Readable.pipe(Writable) only [OK]

Hint: Pipe is always called on readable stream [OK]

Common Mistakes:

• Calling pipe on writable stream
• Ignoring error handling (not main error here)
• Assuming streams can't be piped

5. You want to read data from input.txt, compress it using zlib's gzip, and write the compressed data to output.gz. Which code snippet correctly pipes these streams together?

hard

A. const fs = require('fs');
const zlib = require('zlib');
const readStream = fs.createReadStream('input.txt');
const gzip = zlib.createGzip();
const writeStream = fs.createWriteStream('output.gz');
readStream.pipe(gzip).pipe(writeStream);

B. const fs = require('fs');
const zlib = require('zlib');
const readStream = fs.createReadStream('input.txt');
const gzip = zlib.createGzip();
const writeStream = fs.createWriteStream('output.gz');
gzip.pipe(readStream).pipe(writeStream);

C. const fs = require('fs');
const zlib = require('zlib');
const readStream = fs.createReadStream('input.txt');
const gzip = zlib.createGzip();
const writeStream = fs.createWriteStream('output.gz');
writeStream.pipe(gzip).pipe(readStream);

D. const fs = require('fs');
const zlib = require('zlib');
const readStream = fs.createReadStream('input.txt');
const gzip = zlib.createGzip();
const writeStream = fs.createWriteStream('output.gz');
readStream.pipe(writeStream).pipe(gzip);

Solution

1. Step 1: Understand the stream flow

   Data flows from readable (input.txt) to transform (gzip) to writable (output.gz).

2. Step 2: Check pipe chaining

   const fs = require('fs'); const zlib = require('zlib'); const readStream = fs.createReadStream('input.txt'); const gzip = zlib.createGzip(); const writeStream = fs.createWriteStream('output.gz'); readStream.pipe(gzip).pipe(writeStream); correctly pipes readStream into gzip, then gzip into writeStream. Other options reverse or misuse pipe order.

3. Final Answer:

   readStream.pipe(gzip).pipe(writeStream); -> Option A

4. Quick Check:

   Readable -> Transform -> Writable pipe chain [OK]

Hint: Chain pipes: readable.pipe(transform).pipe(writable) [OK]

Common Mistakes:

• Reversing pipe order
• Calling pipe on writable stream
• Not chaining transform stream correctly