Streaming with ReadableStream and WritableStream in Deno - Deep Dive

Overview - Streaming with ReadableStream and WritableStream

What is it?

Streaming with ReadableStream and WritableStream is a way to handle data piece by piece instead of all at once. ReadableStream lets you read data as it arrives, like watching a video while it downloads. WritableStream lets you send or save data bit by bit, like writing a letter one word at a time. This helps programs work efficiently with large or slow data sources.

Why it matters

Without streaming, programs must wait for all data to load before using it, which can cause delays and use lots of memory. Streaming lets apps start working immediately and handle big files or live data smoothly. This improves user experience and saves resources, especially for web servers, media players, and real-time apps.

Where it fits

Before learning streaming, you should understand basic JavaScript promises and asynchronous programming. After mastering streaming, you can explore advanced data processing, network programming, and building efficient APIs that handle large or continuous data flows.

Mental Model

Core Idea

ReadableStream and WritableStream let programs handle data step-by-step as it flows in or out, avoiding waiting for everything at once.

Think of it like...

Imagine a water pipe: ReadableStream is like water flowing into your house, and WritableStream is like water flowing out to the garden. You don’t need to fill a bucket first; you just use the water as it comes or send it out continuously.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Data Source   │──────▶│ ReadableStream│──────▶│ Your Program  │
└───────────────┘       └───────────────┘       └───────────────┘

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Your Program  │──────▶│ WritableStream│──────▶│ Data Destination│
└───────────────┘       └───────────────┘       └───────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Streams Basics

Concept: Streams let you handle data piece by piece instead of all at once.

A ReadableStream is a source of data you can read from gradually. A WritableStream is a target where you can send data gradually. Think of them as conveyor belts moving data in small chunks.

Result

You can start processing data immediately without waiting for the entire content.

Understanding that streams break data into manageable pieces helps you handle large or slow data sources efficiently.

2

Foundation

Creating a Simple ReadableStream

Concept: You can create a ReadableStream that produces data chunks over time.

Example: Create a ReadableStream that emits numbers 1 to 5 with a delay. const stream = new ReadableStream({ start(controller) { let count = 1; const interval = setInterval(() => { if (count <= 5) { controller.enqueue(count); count++; } else { clearInterval(interval); controller.close(); } }, 500); } });

Result

The stream emits numbers 1 to 5 every 500ms, then closes.

Knowing how to produce data chunks on demand is key to building custom data sources.

3

Intermediate

Consuming Data from ReadableStream

🤔Before reading on: do you think you must wait for all data before reading from a ReadableStream, or can you read chunks as they arrive? Commit to your answer.

Concept: You can read data chunks from a ReadableStream as they become available using a reader.

Use getReader() to read chunks: const reader = stream.getReader(); async function read() { while (true) { const { done, value } = await reader.read(); if (done) break; console.log('Received:', value); } } read();

Result

Numbers 1 to 5 print one by one every 500ms.

Understanding that streams provide data asynchronously lets you process data immediately and efficiently.

4

Intermediate

Writing Data with WritableStream

🤔Before reading on: do you think WritableStream buffers all data internally before writing, or writes chunks as they come? Commit to your answer.

Concept: WritableStream accepts data chunks and processes them as they arrive, often asynchronously.

Example WritableStream that logs chunks: const writable = new WritableStream({ write(chunk) { console.log('Writing:', chunk); }, close() { console.log('Stream closed'); } }); const writer = writable.getWriter(); async function writeData() { for (let i = 1; i <= 3; i++) { await writer.write(i); } await writer.close(); } writeData();

Result

Logs 'Writing: 1', 'Writing: 2', 'Writing: 3', then 'Stream closed'.

Knowing that WritableStream handles chunks as they come helps build responsive data sinks.

5

Intermediate

Piping Data Between Streams

🤔Before reading on: do you think you must manually read and write each chunk to connect streams, or is there a simpler way? Commit to your answer.

Concept: Streams can be connected directly using pipeTo or pipeThrough to transfer data automatically.

Example: Pipe a ReadableStream to a WritableStream: stream.pipeTo(writable).then(() => { console.log('Piping complete'); });

Result

Data flows from the readable to writable stream automatically, logging each chunk.

Understanding piping simplifies connecting data sources and sinks without manual chunk handling.

6

Advanced

Handling Backpressure in Streams

🤔Before reading on: do you think streams always push data as fast as possible, or can they slow down to match the receiver? Commit to your answer.

Concept: Backpressure is when the writable side signals it can't keep up, so the readable side slows down to avoid overload.

Streams use internal signals to pause reading if writing is slow. For example, write() returns a promise that resolves when ready for more data, so producers wait before sending more.

Result

Data flows smoothly without overwhelming the writable stream or causing memory issues.

Knowing backpressure prevents crashes and memory bloat in real-world streaming applications.

7

Expert

Custom Stream Transformations

🤔Before reading on: do you think transforming streams requires manual read-write loops, or can you create reusable transform streams? Commit to your answer.

Concept: TransformStream lets you create streams that modify data chunks as they pass through, combining readable and writable behavior.

Example: TransformStream that doubles numbers: const doubleTransform = new TransformStream({ transform(chunk, controller) { controller.enqueue(chunk * 2); } }); stream.pipeThrough(doubleTransform).pipeTo(writable);

Result

WritableStream receives doubled numbers: 2, 4, 6, 8, 10.

Understanding TransformStream enables building modular, reusable data processing pipelines.

Under the Hood

ReadableStream and WritableStream work by managing internal queues of data chunks and coordinating asynchronous reading and writing. The ReadableStream controller controls when data is enqueued or closed. WritableStream manages backpressure by returning promises from write() calls, signaling when it can accept more data. The event loop and microtasks handle asynchronous chunk delivery without blocking the main thread.

Why designed this way?

Streams were designed to handle large or slow data sources efficiently without blocking or using excessive memory. The promise-based backpressure system ensures smooth flow control. Alternatives like loading entire data at once were inefficient or impossible for big or live data. The design balances simplicity, flexibility, and performance for web and server environments.

┌─────────────────────────────┐
│       ReadableStream         │
│ ┌───────────────┐           │
│ │ Controller    │           │
│ │ (enqueue/close)│          │
│ └───────┬───────┘           │
│         │                   │
│         ▼                   │
│  Internal Queue (chunks)    │
│         │                   │
│         ▼                   │
│      Reader reads chunks    │
└─────────┬───────────────────┘
          │
          ▼
┌─────────────────────────────┐
│      WritableStream          │
│ ┌───────────────┐           │
│ │ Writer        │           │
│ │ (write/close) │           │
│ └───────┬───────┘           │
│         │                   │
│         ▼                   │
│  Internal Queue (chunks)    │
│         │                   │
│         ▼                   │
│   Destination (file/network)│
└─────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do you think ReadableStream always loads all data before you can read any chunk? Commit to yes or no.

Common Belief: ReadableStream waits until all data is available before letting you read.

Reality: ReadableStream provides data chunks as soon as they arrive, allowing immediate processing.

Why it matters: Believing this causes developers to write inefficient code that waits unnecessarily, losing streaming benefits.

Quick: Do you think WritableStream buffers all data internally and writes only at the end? Commit to yes or no.

Common Belief: WritableStream collects all data and writes it only after closing.

Reality: WritableStream writes chunks as they come, often asynchronously, supporting real-time data sinks.

Why it matters: Misunderstanding this leads to incorrect assumptions about latency and resource use in streaming apps.

Quick: Do you think you must manually handle backpressure by pausing and resuming streams? Commit to yes or no.

Common Belief: Backpressure requires manual pausing and resuming of streams by the developer.

Reality: Streams handle backpressure automatically using promises and internal signaling.

Why it matters: Thinking manual control is needed causes complex, error-prone code and bugs.

Quick: Do you think TransformStream is just a fancy name for piping streams? Commit to yes or no.

Common Belief: TransformStream is just a way to connect streams without changing data.

Reality: TransformStream lets you modify data chunks as they pass through, enabling powerful data processing.

Why it matters: Missing this limits the ability to build efficient, modular data pipelines.

Expert Zone

1

Streams use internal queuing strategies that can be customized to optimize memory and latency for specific use cases.

2

WritableStream's write() method returns a promise that resolves only when the chunk is fully processed, enabling precise flow control.

3

TransformStream combines readable and writable sides with a transform function, but its internal buffering and backpressure handling can affect performance subtly.

When NOT to use

Avoid streams when data is small and fits comfortably in memory, as streaming adds complexity. For simple synchronous data, direct arrays or buffers are simpler. Also, for protocols requiring strict message boundaries, specialized parsers may be better than raw streams.

Production Patterns

In production, streams are used for file uploads/downloads, real-time media processing, network communication, and chaining multiple TransformStreams for complex data transformations. They enable scalable servers that handle many simultaneous connections without blocking.

Connections

Reactive Programming

Streams share the idea of handling data over time asynchronously.

Understanding streams helps grasp reactive programming concepts like observables, which also process data as it arrives.

Unix Pipes

Streams in programming are inspired by Unix pipes that connect commands to pass data incrementally.

Knowing Unix pipes clarifies how data flows through streams and why chaining is powerful.

Water Supply Systems

Streams mimic continuous flow systems where supply and demand must be balanced.

Recognizing flow control in water systems helps understand backpressure and buffering in streams.

Common Pitfalls

#1 Trying to read from a ReadableStream without using getReader() or async iteration.

Wrong approach:

const data = stream.read(); // Error: no read() method directly on stream

Correct approach:

const reader = stream.getReader(); const { value, done } = await reader.read();

Root cause:

Misunderstanding that ReadableStream itself is not directly readable without a reader.

#2 Writing to a WritableStream without awaiting the write promise, causing data loss or race conditions.

Wrong approach:

writer.write(chunk); writer.write(nextChunk); // No await

Correct approach:

await writer.write(chunk); await writer.write(nextChunk);

Root cause:

Ignoring that write() is asynchronous and must be awaited to ensure order and completion.

#3 Not handling stream errors, leading to uncaught exceptions and crashes.

Wrong approach:

stream.pipeTo(writable); // No error handling

Correct approach:

stream.pipeTo(writable).catch(e => console.error('Stream error:', e));

Root cause:

Assuming streams never fail or forgetting to catch promise rejections.

Key Takeaways

ReadableStream and WritableStream let you process data piece by piece, improving efficiency and responsiveness.

Streams handle asynchronous data flow with built-in backpressure to prevent overload and memory issues.

You can connect streams easily using pipeTo and pipeThrough, enabling modular data pipelines.

TransformStream allows modifying data on the fly, making streams powerful for real-world processing.

Properly handling asynchronous methods and errors is essential to avoid common streaming bugs.

Practice

1. What is the main purpose of a ReadableStream in Deno?

easy

A. To accept data chunks that can be written

B. To produce data chunks that can be read

C. To store data permanently

D. To convert data into JSON format

Solution

1. Step 1: Understand the role of ReadableStream

   A ReadableStream is designed to produce data chunks that can be read by consumers.

2. Step 2: Differentiate from WritableStream

   A WritableStream accepts data chunks to be written, which is the opposite role.

3. Final Answer:

   To produce data chunks that can be read -> Option B

4. Quick Check:

   ReadableStream produces data = A [OK]

Hint: ReadableStream produces data; WritableStream consumes data [OK]

Common Mistakes:

• Confusing ReadableStream with WritableStream
• Thinking streams store data permanently
• Assuming streams convert data formats automatically

2. Which of the following is the correct way to create a new WritableStream in Deno?

easy

A. const stream = new WritableStream({ write(chunk) { console.log(chunk); } });

B. const stream = WritableStream.write(chunk => console.log(chunk));

C. const stream = new WritableStream.write(chunk => console.log(chunk));

D. const stream = WritableStream({ write(chunk) { console.log(chunk); } });

Solution

1. Step 1: Recall WritableStream constructor syntax

   WritableStream is created with new WritableStream() and an object with a write method.

2. Step 2: Check each option's syntax

   const stream = new WritableStream({ write(chunk) { console.log(chunk); } }); correctly uses new WritableStream({ write(chunk) { ... } }). Others misuse syntax or omit new.

3. Final Answer:

   const stream = new WritableStream({ write(chunk) { console.log(chunk); } }); -> Option A

4. Quick Check:

   WritableStream needs new + write method object = C [OK]

Hint: Use 'new WritableStream' with an object having a write method [OK]

Common Mistakes:

• Omitting 'new' keyword
• Calling WritableStream as a function
• Passing a function directly instead of an object

3. What will the following code output in Deno?

const readable = new ReadableStream({
  start(controller) {
    controller.enqueue('Hello');
    controller.enqueue('World');
    controller.close();
  }
});

const reader = readable.getReader();

(async () => {
  let result = '';
  while(true) {
    const { done, value } = await reader.read();
    if (done) break;
    result += value + ' ';
  }
  console.log(result.trim());
})();

medium

A. Error: reader.read is not a function

B. HelloWorld

C. Hello World

D. undefined

Solution

1. Step 1: Understand the ReadableStream setup

   The stream enqueues two chunks: 'Hello' and 'World', then closes.

2. Step 2: Analyze the reading loop

   The loop reads each chunk, appends it with a space to result, and stops when done is true.

3. Step 3: Output after trimming

   After reading both chunks, result is 'Hello World ' which trims to 'Hello World'.

4. Final Answer:

   Hello World -> Option C

5. Quick Check:

   Chunks read and concatenated with space = D [OK]

Hint: ReadableStream chunks read sequentially, concatenated with spaces [OK]

Common Mistakes:

• Forgetting to await reader.read()
• Not trimming the final string
• Assuming chunks concatenate without spaces

4. Identify the error in this Deno code using streams:

const writable = new WritableStream({
  write(chunk) {
    console.log(chunk.toUpperCase());
  }
});

const writer = writable.getWriter();

await writer.write(123);
await writer.close();

medium

A. write method must be async

B. Missing await before new WritableStream

C. getWriter() is not a function

D. Cannot call toUpperCase on a number

Solution

1. Step 1: Check the write method usage

   The write method calls chunk.toUpperCase(), which only works on strings.

2. Step 2: Analyze the write call argument

   The code writes the number 123, which has no toUpperCase() method, causing a runtime error.

3. Final Answer:

   Cannot call toUpperCase on a number -> Option D

4. Quick Check:

   toUpperCase on number causes error = A [OK]

Hint: Check data types before calling string methods [OK]

Common Mistakes:

• Assuming write must be async
• Thinking getWriter() is invalid
• Missing await on WritableStream creation

5. You want to transform a ReadableStream of text chunks to uppercase and write them to a WritableStream. Which approach correctly implements this in Deno?

hard

A. Pipe the readable through a TransformStream that uppercases chunks, then pipe to writable

B. Directly write chunks from readable to writable without transformation

C. Use readable.getReader() and writable.getWriter(), but forget to release locks

D. Convert chunks to uppercase after writing to writable

Solution

1. Step 1: Understand stream transformation

   To modify data between readable and writable, use a TransformStream that changes chunks to uppercase.

2. Step 2: Use pipeThrough and pipeTo methods

   Pipe the readable stream through the transform, then pipe the result to the writable stream for correct flow and backpressure handling.

3. Step 3: Evaluate other options

   Directly write chunks from readable to writable without transformation skips transformation, C risks locking issues, and D transforms too late after writing.

4. Final Answer:

   Pipe the readable through a TransformStream that uppercases chunks, then pipe to writable -> Option A

5. Quick Check:

   Use TransformStream + pipeThrough + pipeTo = B [OK]

Hint: Use TransformStream between readable and writable for data changes [OK]

Common Mistakes:

• Skipping TransformStream for data modification
• Not handling stream locks properly
• Transforming data after writing