Request/response transformation in Nginx - Deep Dive

Overview - Request/response transformation

What is it?

Request/response transformation is the process of changing the data or headers in HTTP requests or responses as they pass through a server like nginx. This lets you modify what clients send or receive without changing the backend application. For example, you can add headers, rewrite URLs, or change content on the fly.

Why it matters

Without request/response transformation, you would need to change every backend service to handle different client needs or security rules. This would be slow, error-prone, and hard to maintain. Transformations let you centralize changes, improve security, and adapt traffic flexibly, saving time and reducing bugs.

Where it fits

Before learning this, you should understand basic HTTP concepts and how nginx works as a reverse proxy. After this, you can explore advanced nginx modules, caching strategies, and API gateways that build on transformation techniques.

Mental Model

Core Idea

Request/response transformation is like a smart mail sorter that changes letters as they pass through, so the sender and receiver get exactly what they need without changing the original sender or receiver.

Think of it like...

Imagine a postal worker who opens letters, adds or removes pages, or changes the envelope address before sending them on. The sender and receiver don’t know about these changes, but the mail fits their needs perfectly.

Client ──▶ [nginx Transformer] ──▶ Backend Server
  ▲                             ▼
  │                             │
  ◀───────── Response ──────────

Build-Up - 7 Steps

1

Foundation

Understanding HTTP requests and responses

Concept: Learn the basic structure of HTTP requests and responses, including headers and body.

HTTP requests have a method (GET, POST), URL, headers, and optional body. Responses have a status code, headers, and body. nginx can read and modify these parts.

Result

You can identify parts of HTTP messages that can be changed during transformation.

Understanding HTTP message parts is essential because transformations target these exact parts.

2

Foundation

Basic nginx proxying setup

Concept: Set up nginx as a reverse proxy to forward requests to a backend server.

A simple nginx config uses 'proxy_pass' to send client requests to backend. Example: server { listen 80; location / { proxy_pass http://backend_server; } }

Result

nginx forwards requests and responses without changes.

Knowing how proxying works is the base for adding transformations.

3

Intermediate

Modifying request headers with proxy_set_header

🤔Before reading on: do you think changing request headers affects only nginx or the backend server? Commit to your answer.

Concept: Use 'proxy_set_header' to add, change, or remove headers sent to the backend.

Example: location / { proxy_pass http://backend_server; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header Host $host; } This adds or overrides headers before sending the request.

Result

Backend server receives modified headers, influencing its behavior.

Knowing that headers can be changed on the fly lets you control backend behavior without touching backend code.

4

Intermediate

Rewriting request URLs with rewrite directive

🤔Before reading on: do you think rewriting URLs changes what the client sees or only what the backend receives? Commit to your answer.

Concept: Use 'rewrite' to change the request URL path before proxying.

Example: location /oldpath/ { rewrite ^/oldpath/(.*)$ /newpath/$1 break; proxy_pass http://backend_server; } This changes requests from /oldpath/... to /newpath/... internally.

Result

Backend receives requests with rewritten URLs, while clients see original URLs.

Understanding URL rewriting helps adapt legacy clients or APIs without backend changes.

5

Intermediate

Adding response headers with add_header

🤔Before reading on: do you think 'add_header' can modify all responses or only successful ones? Commit to your answer.

Concept: Use 'add_header' to insert headers into responses sent back to clients.

Example: location / { proxy_pass http://backend_server; add_header X-Custom-Header "MyValue"; } This adds a custom header to the response.

Result

Clients receive extra headers that can control caching, security, or info.

Knowing how to add headers on responses lets you enhance client experience or enforce policies.

6

Advanced

Transforming response body with sub_filter module

🤔Before reading on: do you think nginx can change the actual content of a response body? Commit to your answer.

Concept: Use 'sub_filter' to search and replace text in response bodies passing through nginx.

Example: location / { proxy_pass http://backend_server; sub_filter 'oldtext' 'newtext'; sub_filter_once off; } This replaces all occurrences of 'oldtext' with 'newtext' in responses.

Result

Clients see modified content without backend changes.

Understanding body transformation opens powerful ways to fix or customize content dynamically.

7

Expert

Handling transformation performance and limitations

🤔Before reading on: do you think heavy transformations always improve user experience? Commit to your answer.

Concept: Learn how transformations affect performance and when they can cause issues like buffering or delays.

Transformations like sub_filter require buffering full responses, increasing memory and latency. Overusing transformations can slow nginx and degrade user experience. Use caching and limit transformations to critical cases.

Result

You can balance transformation benefits with system performance.

Knowing transformation costs prevents production slowdowns and helps design efficient systems.

Under the Hood

nginx processes requests and responses through a chain of modules. When configured, transformation modules intercept HTTP messages, modify headers or body content in memory buffers, then pass them along. For body changes, nginx buffers the entire response to apply substitutions before sending to the client.

Why designed this way?

nginx was designed for high performance and flexibility. Transformations are optional modules to avoid overhead when not needed. Buffering is necessary for body changes but kept minimal to maintain speed. This modular design balances power and efficiency.

Client Request
   │
   ▼
[nginx core]
   │
   ├─▶ Header transformation modules (modify headers)
   │
   ├─▶ URL rewrite module (change request path)
   │
   ├─▶ Proxy module (forwards request)
   │
   ▼
Backend Server
   │
   ▲
Response
   │
   ├─▶ Buffer response body (if needed)
   │
   ├─▶ Body transformation modules (e.g., sub_filter)
   │
   ├─▶ Add response headers
   │
   ▼
Client Response

Myth Busters - 4 Common Misconceptions

Quick: Does 'proxy_set_header' change headers seen by the client? Commit yes or no.

Common Belief: proxy_set_header changes headers that the client receives.

Reality: proxy_set_header only changes headers sent to the backend server, not the client.

Why it matters: Misunderstanding this leads to failed attempts to control client-visible headers, causing security or functionality issues.

Quick: Can nginx transform response bodies without buffering the entire content? Commit yes or no.

Common Belief: nginx can stream and transform response bodies without buffering all data.

Reality: For body transformations like sub_filter, nginx must buffer the full response before sending it to the client.

Why it matters: Ignoring buffering needs can cause unexpected delays or memory exhaustion in production.

Quick: Does rewriting URLs with 'rewrite' change what the client sees in the browser address bar? Commit yes or no.

Common Belief: URL rewriting changes the URL the client sees in their browser.

Reality: nginx URL rewriting changes the internal request path sent to the backend but does not change the client's visible URL.

Why it matters: Confusing this can cause broken links or user confusion when URLs don’t match expectations.

Quick: Can 'add_header' add headers to all response codes by default? Commit yes or no.

Common Belief: add_header adds headers to every response regardless of status code.

Reality: By default, add_header only adds headers to successful (2xx) responses unless configured otherwise.

Why it matters: This can cause missing headers on error pages, leading to inconsistent client behavior.

Expert Zone

1

Transformations can interfere with caching headers, so you must carefully coordinate them to avoid stale or incorrect cached content.

2

Using variables in proxy_set_header or rewrite directives can introduce subtle bugs if variable values are unexpected or empty.

3

Some transformations require enabling buffering, which can increase memory use and latency; experts tune buffer sizes and limits to balance performance.

When NOT to use

Avoid heavy response body transformations on large or streaming content; instead, use backend changes or dedicated content transformation services. For simple header changes, prefer lightweight modules to minimize overhead.

Production Patterns

In production, nginx often adds security headers (CSP, HSTS) via add_header, rewrites legacy API paths to new ones, and uses sub_filter sparingly to fix minor content issues without backend redeploys.

Connections

API Gateway

Request/response transformation is a core feature of API gateways that manage and modify traffic between clients and services.

Understanding nginx transformations helps grasp how API gateways enforce policies, modify requests, and shape responses in microservice architectures.

Content Delivery Network (CDN)

CDNs often perform similar transformations to optimize or secure content delivery closer to users.

Knowing nginx transformations clarifies how edge servers modify traffic for performance and security.

Postal Mail Sorting

Both involve intercepting and modifying messages in transit to meet recipient needs without changing sender or receiver.

Recognizing this pattern across domains reveals a universal approach to message adaptation and control.

Common Pitfalls

#1 Trying to add a header to all responses but it only appears on 200 OK.

Wrong approach:

add_header X-Test "value";

Correct approach:

add_header X-Test "value" always;

Root cause:

By default, add_header applies only to successful responses; missing 'always' causes headers to be skipped on errors.

#2 Expecting proxy_set_header to change client-visible headers.

Wrong approach:

proxy_set_header X-Custom "value";

Correct approach:

add_header X-Custom "value";

Root cause:

proxy_set_header modifies headers sent to backend, not to clients; add_header is needed for client headers.

#3 Using sub_filter without enabling response buffering.

Wrong approach:

location / {
proxy_pass http://backend;
sub_filter 'foo' 'bar';
}

Correct approach:

location / {
proxy_pass http://backend;
proxy_buffering on;
sub_filter 'foo' 'bar';
sub_filter_once off;
}

Root cause:

sub_filter requires buffering to modify response body; missing buffering causes no changes or errors.

Key Takeaways

Request/response transformation lets nginx modify HTTP traffic without changing backend or client code.

Headers and URLs can be changed separately for requests and responses using specific nginx directives.

Body transformations require buffering and can impact performance, so use them carefully.

Misunderstanding which directives affect client vs backend can cause bugs and security issues.

Expert use balances transformation power with system efficiency and maintainability.