Buffer and streams relationship in Node.js

Introduction

Buffers temporarily hold data in memory. Streams let data flow in small parts. Together, they help handle big data smoothly without waiting for everything at once.

Reading a large file piece by piece instead of loading it all at once.

Sending data over the internet in chunks to avoid delays.

Processing video or audio streams in real time.

Handling user uploads without freezing the app.

Transforming data on the fly, like compressing or encrypting.

Syntax

Node.js

const stream = require('stream');

// Create a readable stream
const readable = new stream.Readable({
  read(size) {
    // Push data as Buffer chunks
    this.push(Buffer.from('Hello'));
    this.push(null); // No more data
  }
});

Buffers hold raw binary data temporarily.

Streams use Buffers internally to send data in chunks.

Examples

This creates a Buffer holding the text 'Hello'.

Node.js

const buf = Buffer.from('Hello');
console.log(buf);

A readable stream pushes Buffer chunks to be read.

Node.js

const { Readable } = require('stream');

const readable = new Readable({
  read() {
    this.push(Buffer.from('Data chunk'));
    this.push(null);
  }
});

Listening to stream data events to get Buffer chunks and convert to string.

Node.js

readable.on('data', (chunk) => {
  console.log('Received:', chunk.toString());
});

Sample Program

This program creates a readable stream that sends three Buffer chunks: 'Hello ', 'World', and '!'. It prints each chunk as it arrives and then prints a message when all data is done.

Node.js

const { Readable } = require('stream');

// Create a readable stream that sends 3 chunks
const readable = new Readable({
  read() {
    this.push(Buffer.from('Hello '));
    this.push(Buffer.from('World'));
    this.push(Buffer.from('!'));
    this.push(null); // No more data
  }
});

// Listen for data events
readable.on('data', (chunk) => {
  console.log('Chunk received:', chunk.toString());
});

// Listen for end event
readable.on('end', () => {
  console.log('No more data.');
});

Important Notes

Streams use Buffers internally to handle data efficiently.

Buffers let streams send data in small pieces, saving memory.

Always listen for 'end' event to know when stream finishes.

Summary

Buffers temporarily hold raw data in memory.

Streams send data in chunks using Buffers.

Together, they help process large data smoothly and efficiently.