Buffer allocation and encoding in Node.js

Introduction

Buffers let you work with raw binary data in Node.js. You use them to store and manipulate bytes directly.

When reading or writing files with binary data like images or videos.

When sending or receiving data over a network socket.

When converting strings to bytes for encoding or decoding.

When working with streams that handle raw data.

When you need to manipulate data at the byte level for performance.

Syntax

Node.js

const buf = Buffer.alloc(size, fill, encoding);
const bufFromString = Buffer.from(string, encoding);

Buffer.alloc(size) creates a buffer of fixed size filled with zeros by default.

Buffer.from() creates a buffer from a string or array, using the specified encoding.

Examples

Creates a buffer of 5 bytes, all set to zero.

Node.js

const buf = Buffer.alloc(5);

Creates a buffer of 5 bytes, each byte set to the ASCII code for 'a'.

Node.js

const buf = Buffer.alloc(5, 'a');

Creates a buffer from the string 'hello' using UTF-8 encoding.

Node.js

const buf = Buffer.from('hello', 'utf8');

Creates a buffer from an array of bytes.

Node.js

const buf = Buffer.from([1, 2, 3, 4]);

Sample Program

This program shows how to allocate buffers with zeros or a specific character, and how to create a buffer from a string. It prints the buffers to see their contents.

Node.js

import { Buffer } from 'node:buffer';

// Allocate a buffer of 10 bytes, filled with zeros
const zeroBuffer = Buffer.alloc(10);

// Allocate a buffer of 10 bytes, filled with 'x' (ASCII 120)
const xBuffer = Buffer.alloc(10, 'x');

// Create a buffer from a string with UTF-8 encoding
const helloBuffer = Buffer.from('hello', 'utf8');

// Show the buffers as strings and arrays
console.log('zeroBuffer:', zeroBuffer);
console.log('xBuffer as string:', xBuffer.toString());
console.log('helloBuffer as array:', helloBuffer);
console.log('helloBuffer as string:', helloBuffer.toString());

Important Notes

Buffers are fixed size. You cannot resize them after creation.

Always specify encoding when converting strings to buffers to avoid confusion.

Use Buffer.allocUnsafe(size) only if you plan to overwrite all bytes, as it may contain old data.

Summary

Buffers store raw binary data in Node.js.

Use Buffer.alloc() to create zero-filled buffers.

Use Buffer.from() to create buffers from strings or arrays with encoding.

Practice

1. What does Buffer.alloc(5) do in Node.js?

easy

A. Creates a buffer of length 5 filled with zeros

B. Creates a buffer of length 5 filled with random data

C. Creates a buffer from a string of length 5

D. Allocates memory but does not initialize the buffer

Solution

1. Step 1: Understand Buffer.alloc behavior

   Buffer.alloc(size) creates a buffer of the given size and fills it with zeros for safety.

2. Step 2: Apply to size 5

   Calling Buffer.alloc(5) creates a buffer of length 5 filled with zeros.

3. Final Answer:

   Creates a buffer of length 5 filled with zeros -> Option A

4. Quick Check:

   Buffer.alloc(5) = zero-filled buffer [OK]

Hint: Buffer.alloc always zero-fills the buffer size [OK]

Common Mistakes:

• Thinking Buffer.alloc fills with random data
• Confusing Buffer.alloc with Buffer.from
• Assuming buffer is uninitialized

2. Which of the following is the correct syntax to create a buffer from the string 'hello' using UTF-8 encoding?

easy

A. Buffer.alloc('hello', 'utf8')

B. Buffer.create('hello', 'utf8')

C. new Buffer('hello', 'utf8')

D. Buffer.from('hello', 'utf8')

Solution

1. Step 1: Identify correct Buffer creation method

   In modern Node.js, Buffer.from() creates a buffer from a string with encoding.

2. Step 2: Check syntax correctness

   Buffer.from('hello', 'utf8') is correct syntax; Buffer.alloc does not accept string input, and Buffer.create does not exist. new Buffer() is deprecated.

3. Final Answer:

   Buffer.from('hello', 'utf8') -> Option D

4. Quick Check:

   Use Buffer.from for string buffers [OK]

Hint: Use Buffer.from for strings, Buffer.alloc for size [OK]

Common Mistakes:

• Using Buffer.alloc with string input
• Using deprecated new Buffer()
• Using non-existent Buffer.create()

3. What will be the output of the following code?

const buf = Buffer.from('abc', 'utf8');
console.log(buf.length);

medium

A. 3

B. 6

C. 1

D. Error

Solution

1. Step 1: Understand Buffer.from with UTF-8 string

   The string 'abc' has 3 characters, each encoded as 1 byte in UTF-8.

2. Step 2: Check buffer length property

   The buffer length equals the number of bytes, so buf.length is 3.

3. Final Answer:

   3 -> Option A

4. Quick Check:

   UTF-8 'abc' length = 3 bytes [OK]

Hint: Buffer length equals byte count of string encoding [OK]

Common Mistakes:

• Assuming length is character count for multi-byte chars
• Expecting length to be string length property
• Confusing bytes with characters

4. Identify the error in this code snippet:

const buf = Buffer.alloc(5, 'abc');
console.log(buf.toString());

medium

A. Buffer.alloc cannot take a string as fill

B. toString() is not a Buffer method

C. No error, prints 'abcab'

D. Buffer.alloc size must be a string

Solution

1. Step 1: Check Buffer.alloc parameters

   Buffer.alloc(size, fill) accepts a size and a fill value. The fill can be a string, which repeats to fill the buffer.

2. Step 2: Understand fill behavior and toString()

   Filling 5 bytes with 'abc' repeats 'abcab'. toString() converts buffer back to string.

3. Final Answer:

   No error, prints 'abcab' -> Option C

4. Quick Check:

   Buffer.alloc fills string repeatedly [OK]

Hint: Buffer.alloc fills string repeatedly if fill is string [OK]

Common Mistakes:

• Thinking Buffer.alloc fill cannot be string
• Assuming toString() is invalid on Buffer
• Expecting error due to fill length mismatch

5. You want to create a buffer from the string 'café' and then convert it back to a string. Which encoding should you use to preserve the accented character correctly?

hard

A. 'ascii'

B. 'utf8'

C. 'base64'

D. 'hex'

Solution

1. Step 1: Understand character encoding for accented characters

   ASCII encoding cannot represent accented characters like 'é'. UTF-8 supports all Unicode characters including accented ones.

2. Step 2: Choose encoding for buffer creation and conversion

   Using 'utf8' ensures the accented character is preserved when converting to and from buffer.

3. Final Answer:

   'utf8' -> Option B

4. Quick Check:

   Use UTF-8 for accented characters [OK]

Hint: Use UTF-8 to handle accented characters correctly [OK]

Common Mistakes:

• Using ASCII which drops accents
• Using base64 or hex which are encoding formats, not text encodings
• Confusing base64 with UTF-8