Piping streams together in Node.js - Mini Project: Build & Apply

Piping Streams Together in Node.js

📖 Scenario: You are building a simple Node.js script that reads data from one file, transforms it, and writes the result to another file using streams.This technique is useful for handling large files efficiently without loading everything into memory.

🎯 Goal: Create a Node.js script that pipes a readable stream from input.txt through a transform stream that converts all text to uppercase, and then pipes it to a writable stream to output.txt.

📋 What You'll Learn

Create a readable stream from input.txt

Create a writable stream to output.txt

Create a transform stream that converts input text to uppercase

Pipe the streams together in the correct order

💡 Why This Matters

🌍 Real World

Piping streams is used in real-world Node.js applications to process large files, such as logs or media files, efficiently without using too much memory.

💼 Career

Understanding streams and piping is essential for backend developers working with Node.js to build scalable and performant data processing applications.

1

Create a readable stream from input.txt

Write a line of code to create a readable stream called readStream from the file input.txt using fs.createReadStream.

Node.js

import fs from 'fs';
// Create a readable stream from 'input.txt'
// Your code here

Hint

Use fs.createReadStream('input.txt') and assign it to readStream.

2

Create a writable stream to output.txt

Add a line of code to create a writable stream called writeStream to the file output.txt using fs.createWriteStream.

Node.js

import fs from 'fs';
const readStream = fs.createReadStream('input.txt');
// Create a writable stream to 'output.txt'
// Your code here

Hint

Use fs.createWriteStream('output.txt') and assign it to writeStream.

3

Create a transform stream to convert text to uppercase

Write code to import Transform from stream and create a transform stream called upperCaseTransform that converts all input chunks to uppercase strings.

Node.js

import fs from 'fs';
const readStream = fs.createReadStream('input.txt');
const writeStream = fs.createWriteStream('output.txt');
// Import Transform and create upperCaseTransform stream
// Your code here

Hint

Create a Transform stream with a transform method that converts chunks to uppercase.

4

Pipe the streams together

Use the pipe method to connect readStream to upperCaseTransform, and then pipe upperCaseTransform to writeStream.

Node.js

import fs from 'fs';
import { Transform } from 'stream';
const readStream = fs.createReadStream('input.txt');
const writeStream = fs.createWriteStream('output.txt');

const upperCaseTransform = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  }
});

// Pipe readStream to upperCaseTransform, then to writeStream
// Your code here

Hint

Use readStream.pipe(upperCaseTransform).pipe(writeStream) to connect the streams.

Practice

1. What is the main purpose of piping streams together in Node.js?

easy

A. To pause and resume streams manually

B. To convert streams into arrays for easier processing

C. To connect a readable stream directly to a writable stream for automatic data flow

D. To create new streams from scratch

Solution

1. Step 1: Understand what piping does

   Piping connects a readable stream to a writable stream so data flows automatically without manual intervention.

2. Step 2: Compare options

   Only To connect a readable stream directly to a writable stream for automatic data flow describes this automatic connection and data flow. Other options describe unrelated stream operations.

3. Final Answer:

   To connect a readable stream directly to a writable stream for automatic data flow -> Option C

4. Quick Check:

   Piping = automatic stream connection [OK]

Hint: Piping means connecting streams for automatic data transfer [OK]

Common Mistakes:

• Thinking piping converts streams to arrays
• Confusing piping with manual pause/resume
• Assuming piping creates new streams

2. Which of the following is the correct syntax to pipe a readable stream readStream into a writable stream writeStream?

easy

A. readStream.pipe(writeStream);

B. writeStream.pipe(readStream);

C. pipe(readStream, writeStream);

D. readStream.write(writeStream);

Solution

1. Step 1: Recall pipe method usage

   The pipe() method is called on a readable stream and takes a writable stream as argument.

2. Step 2: Check each option

   readStream.pipe(writeStream); matches the correct syntax. writeStream.pipe(readStream); reverses streams, C uses a non-existent function, D misuses write method.

3. Final Answer:

   readStream.pipe(writeStream); -> Option A

4. Quick Check:

   Readable.pipe(Writable) = correct syntax [OK]

Hint: Remember: readableStream.pipe(writableStream) [OK]

Common Mistakes:

• Reversing the order of streams in pipe
• Using pipe as a standalone function
• Calling write instead of pipe

3. Consider this code snippet:

const fs = require('fs');
const readStream = fs.createReadStream('input.txt');
const writeStream = fs.createWriteStream('output.txt');
readStream.pipe(writeStream);
writeStream.on('finish', () => console.log('Done'));

What will be printed when the piping finishes?

medium

A. No output

B. Error

C. undefined

D. Done

Solution

1. Step 1: Understand the pipe and event

   The readable stream pipes data to the writable stream. When writing finishes, the 'finish' event triggers.

2. Step 2: Check the event handler

   The code listens for 'finish' on writeStream and logs 'Done' when triggered.

3. Final Answer:

   Done -> Option D

4. Quick Check:

   'finish' event logs 'Done' [OK]

Hint: Listen to 'finish' event on writable stream for completion [OK]

Common Mistakes:

• Expecting 'end' event on writable stream
• Not handling asynchronous event
• Confusing 'finish' with 'close'

4. What is wrong with this code snippet?

const fs = require('fs');
const readStream = fs.createReadStream('file.txt');
const writeStream = fs.createWriteStream('copy.txt');
writeStream.pipe(readStream);

medium

A. The file paths are incorrect

B. The pipe method is called on the writable stream instead of the readable stream

C. Missing error handling on streams

D. Streams cannot be piped in Node.js

Solution

1. Step 1: Identify pipe usage

   The pipe method must be called on a readable stream, passing a writable stream as argument.

2. Step 2: Analyze the code

   The code calls pipe on writeStream (writable), which is incorrect and will cause an error.

3. Final Answer:

   The pipe method is called on the writable stream instead of the readable stream -> Option B

4. Quick Check:

   Readable.pipe(Writable) only [OK]

Hint: Pipe is always called on readable stream [OK]

Common Mistakes:

• Calling pipe on writable stream
• Ignoring error handling (not main error here)
• Assuming streams can't be piped

5. You want to read data from input.txt, compress it using zlib's gzip, and write the compressed data to output.gz. Which code snippet correctly pipes these streams together?

hard

A. const fs = require('fs');
const zlib = require('zlib');
const readStream = fs.createReadStream('input.txt');
const gzip = zlib.createGzip();
const writeStream = fs.createWriteStream('output.gz');
readStream.pipe(gzip).pipe(writeStream);

B. const fs = require('fs');
const zlib = require('zlib');
const readStream = fs.createReadStream('input.txt');
const gzip = zlib.createGzip();
const writeStream = fs.createWriteStream('output.gz');
gzip.pipe(readStream).pipe(writeStream);

C. const fs = require('fs');
const zlib = require('zlib');
const readStream = fs.createReadStream('input.txt');
const gzip = zlib.createGzip();
const writeStream = fs.createWriteStream('output.gz');
writeStream.pipe(gzip).pipe(readStream);

D. const fs = require('fs');
const zlib = require('zlib');
const readStream = fs.createReadStream('input.txt');
const gzip = zlib.createGzip();
const writeStream = fs.createWriteStream('output.gz');
readStream.pipe(writeStream).pipe(gzip);

Solution

1. Step 1: Understand the stream flow

   Data flows from readable (input.txt) to transform (gzip) to writable (output.gz).

2. Step 2: Check pipe chaining

   const fs = require('fs'); const zlib = require('zlib'); const readStream = fs.createReadStream('input.txt'); const gzip = zlib.createGzip(); const writeStream = fs.createWriteStream('output.gz'); readStream.pipe(gzip).pipe(writeStream); correctly pipes readStream into gzip, then gzip into writeStream. Other options reverse or misuse pipe order.

3. Final Answer:

   readStream.pipe(gzip).pipe(writeStream); -> Option A

4. Quick Check:

   Readable -> Transform -> Writable pipe chain [OK]

Hint: Chain pipes: readable.pipe(transform).pipe(writable) [OK]

Common Mistakes:

• Reversing pipe order
• Calling pipe on writable stream
• Not chaining transform stream correctly