Streaming and chunked transfer in Nginx - Commands & Configuration

Introduction

Sometimes, web servers need to send data to browsers or clients as it is generated, without waiting for the whole content to be ready. Streaming and chunked transfer let nginx send data in pieces, so users start receiving content faster and the server uses memory efficiently.

When serving live video or audio streams that are generated on the fly.

When delivering large files without loading them fully into memory first.

When proxying requests to backend servers that send data slowly or in parts.

When you want to reduce latency so users see content as soon as possible.

When handling APIs that send data in multiple parts or events.

Config File - nginx.conf

nginx.conf

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    server {
        listen 8080;

        location /stream {
            # Enable chunked transfer encoding
            chunked_transfer_encoding on;

            # Disable buffering to send data immediately
            proxy_buffering off;

            # Proxy to a backend that streams data
            proxy_pass http://localhost:9000;
        }
    }
}

This configuration sets up an nginx server listening on port 8080.

The /stream location enables chunked_transfer_encoding on; so nginx sends data in chunks.

proxy_buffering off; disables buffering so data is sent immediately as it arrives from the backend.

proxy_pass forwards requests to a backend server on port 9000 that streams data.

Commands

Check the nginx configuration file for syntax errors before starting or reloading nginx.

Terminal

nginx -t

Expected Output

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok

nginx: configuration file /etc/nginx/nginx.conf test is successful

Reload nginx to apply the new configuration without stopping the server.

Terminal

systemctl reload nginx

Expected Output

No output (command runs silently)

Send a request to the streaming endpoint to verify that data is received in chunks.

Terminal

curl -i http://localhost:8080/stream

Expected Output

HTTP/1.1 200 OK

Server: nginx/1.24.0

Date: Thu, 01 Jun 2024 12:00:00 GMT

Transfer-Encoding: chunked

Content-Type: text/plain



4

Wiki

5

pedia

E

 in


chunks.

0

Key Concept

If you remember nothing else from this pattern, remember: enabling chunked transfer and disabling buffering lets nginx send data to clients immediately in small pieces as it arrives.

Common Mistakes

Not disabling proxy_buffering when expecting streaming data.

Nginx buffers the entire response before sending, causing delays and defeating streaming.

Set proxy_buffering off; in the location block to send data immediately.

Forgetting to enable chunked_transfer_encoding.

Without chunked transfer encoding, nginx may not send data in chunks, causing clients to wait.

Add chunked_transfer_encoding on; to enable chunked data sending.

Testing with curl without the -i flag to see headers.

You miss seeing the Transfer-Encoding header that confirms chunked transfer.

Use curl -i to see HTTP headers and verify chunked transfer encoding.

Summary

Configure nginx with chunked_transfer_encoding on and proxy_buffering off to enable streaming.

Reload nginx after configuration changes to apply them without downtime.

Test the streaming endpoint with curl -i to confirm chunked transfer encoding is active.

Practice

1. What is the main purpose of enabling chunked transfer encoding in nginx streaming?

easy

A. To block the client from receiving partial data

B. To compress all data before sending it to the client

C. To send data in smaller parts so the client can start processing early

D. To cache the entire response before sending it

Solution

1. Step 1: Understand chunked transfer encoding

   Chunked transfer encoding breaks the response into smaller pieces called chunks.

2. Step 2: Identify the benefit in streaming

   This allows the client to start receiving and processing data immediately without waiting for the full response.

3. Final Answer:

   To send data in smaller parts so the client can start processing early -> Option C

4. Quick Check:

   Chunked transfer = send data in parts [OK]

Hint: Chunked transfer means sending data in parts, not all at once [OK]

Common Mistakes:

• Thinking chunked transfer compresses data
• Assuming chunked transfer caches full response
• Believing chunked transfer blocks partial data

2. Which nginx directive disables response buffering to enable streaming with chunked transfer?

easy

A. proxy_buffering off;

B. chunked_transfer on;

C. streaming_enable yes;

D. buffer_response off;

Solution

1. Step 1: Identify the directive for buffering control

   In nginx, proxy_buffering controls whether responses are buffered.

2. Step 2: Find the correct syntax to disable buffering

   Setting proxy_buffering off; disables buffering, allowing streaming.

3. Final Answer:

   proxy_buffering off; -> Option A

4. Quick Check:

   Disable buffering = proxy_buffering off [OK]

Hint: Disable buffering with 'proxy_buffering off;' in nginx config [OK]

Common Mistakes:

• Using non-existent directives like chunked_transfer
• Confusing buffering with streaming_enable
• Wrong syntax like buffer_response off

3. Given this nginx config snippet:

location /stream {
  proxy_pass http://backend;
  proxy_buffering off;
  chunked_transfer_encoding on;
}

What will happen when a client requests /stream?

medium

A. Client receives the full response only after backend finishes sending

B. Client receives data in chunks as backend sends it

C. Client connection will be refused due to syntax error

D. Client receives compressed data without chunking

Solution

1. Step 1: Analyze proxy_buffering off;

   This disables buffering, so nginx forwards data immediately.

2. Step 2: Analyze chunked_transfer_encoding on;

   This enables chunked transfer encoding, sending data in parts.

3. Final Answer:

   Client receives data in chunks as backend sends it -> Option B

4. Quick Check:

   Buffering off + chunked on = streaming chunks [OK]

Hint: Buffering off + chunked on means streaming chunks to client [OK]

Common Mistakes:

• Assuming full response is buffered before sending
• Thinking config has syntax errors
• Confusing chunked transfer with compression

4. You configured nginx with:

proxy_buffering off;
chunked_transfer_encoding off;

But streaming is slow and client waits for full response. What is the likely issue?

medium

A. Missing gzip on directive slows down streaming

B. proxy_buffering off causes buffering and delays streaming

C. proxy_pass directive is missing, so no backend connection

D. chunked_transfer_encoding off disables chunked streaming, causing delay

Solution

1. Step 1: Understand effect of chunked_transfer_encoding off;

   Disabling chunked transfer means data is sent as one block, not in parts.

2. Step 2: Analyze impact on streaming

   Without chunked transfer, client waits for full response before receiving data, causing delay.

3. Final Answer:

   chunked_transfer_encoding off disables chunked streaming, causing delay -> Option D

4. Quick Check:

   Chunked off = no streaming chunks [OK]

Hint: Chunked transfer must be on for streaming, else client waits [OK]

Common Mistakes:

• Thinking proxy_buffering off causes buffering
• Assuming gzip affects chunked streaming
• Ignoring missing proxy_pass as cause

5. You want to stream a live video feed through nginx using chunked transfer. Which combination of directives correctly enables streaming with chunked transfer and disables buffering?

hard

A. proxy_buffering off; chunked_transfer_encoding on;

B. proxy_buffering on; chunked_transfer_encoding off;

C. proxy_buffering off; chunked_transfer_encoding off;

D. proxy_buffering on; chunked_transfer_encoding on;

Solution

1. Step 1: Disable buffering for immediate data forwarding

   Set proxy_buffering off; to prevent nginx from waiting to buffer full response.

2. Step 2: Enable chunked transfer encoding for partial data sending

   Set chunked_transfer_encoding on; to send data in chunks to the client.

3. Final Answer:

   proxy_buffering off; chunked_transfer_encoding on; -> Option A

4. Quick Check:

   Disable buffering + enable chunked = streaming [OK]

Hint: Disable buffering and enable chunked transfer for streaming [OK]

Common Mistakes:

• Enabling buffering blocks streaming
• Disabling chunked transfer stops chunked sending
• Mixing buffering and chunked transfer settings incorrectly