Streams vs loading entire file in memory in Node.js

Introduction

Streams let you handle data bit by bit instead of all at once. This helps your program use less memory and work faster with big files.

Reading or writing large files without freezing your app

Processing data as it arrives, like downloading or uploading files

Working with real-time data like video or audio

Avoiding crashes when memory is limited

Sending data over the internet in chunks

Syntax

Node.js

const fs = require('fs');

// Using stream to read file
const stream = fs.createReadStream('file.txt');
stream.on('data', chunk => {
  console.log('Received chunk:', chunk.toString());
});

// Reading entire file at once
fs.readFile('file.txt', (err, data) => {
  if (err) throw err;
  console.log('File content:', data.toString());
});

Streams use events like 'data' to handle chunks of data as they come.

Reading entire file loads all data into memory before you can use it.

Examples

This reads the file piece by piece and logs each chunk size.

Node.js

const fs = require('fs');

// Stream example
const readStream = fs.createReadStream('bigfile.txt');
readStream.on('data', chunk => {
  console.log('Chunk size:', chunk.length);
});

This reads the entire file at once and logs its length.

Node.js

const fs = require('fs');

// Read whole file
fs.readFile('smallfile.txt', (err, data) => {
  if (err) throw err;
  console.log('File length:', data.length);
});

Sample Program

This program shows both ways: streaming the file in parts and reading it all at once. You see chunks printed as they arrive, then the full content after reading all at once.

Node.js

const fs = require('fs');

// Stream reading example
console.log('Start streaming file...');
const stream = fs.createReadStream('example.txt');
stream.on('data', chunk => {
  console.log('Stream chunk:', chunk.toString());
});
stream.on('end', () => {
  console.log('Finished streaming file.');
});

// Read entire file example
fs.readFile('example.txt', (err, data) => {
  if (err) throw err;
  console.log('Read entire file content:', data.toString());
});

Important Notes

Streams are better for big files to save memory.

Reading whole file is simpler but can crash if file is too big.

Use streams when you want to start processing data immediately.

Summary

Streams let you handle data in small pieces, saving memory.

Loading entire file reads all data at once, which can be slow or crash for big files.

Choose streams for big or continuous data, whole file reading for small files.