sendfile and tcp_nopush in Nginx - Deep Dive

Overview - sendfile and tcp_nopush

What is it?

sendfile and tcp_nopush are settings used in nginx to optimize how files are sent over the network. sendfile allows nginx to send files directly from disk to the network without copying data between user space and kernel space. tcp_nopush is a TCP socket option that controls how data packets are grouped before sending, reducing network overhead.

Why it matters

Without these optimizations, nginx would use more CPU and memory to send files, and network packets might be sent inefficiently. This can slow down websites, increase server load, and cause delays for users. Using sendfile and tcp_nopush helps servers deliver files faster and handle more users smoothly.

Where it fits

Learners should first understand basic nginx configuration and how HTTP servers send files. After this, they can explore advanced performance tuning and network socket options to improve server efficiency.

Mental Model

Core Idea

sendfile and tcp_nopush work together to send files efficiently by minimizing data copying and grouping network packets for faster delivery.

Think of it like...

Imagine sending a package: sendfile is like handing the package directly from the warehouse to the delivery truck without unpacking it, and tcp_nopush is like waiting to fill the truck fully before driving off, so fewer trips are needed.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   Disk File   │──────▶│  sendfile()   │──────▶│  Network Socket│
└───────────────┘       └───────────────┘       └───────────────┘
                                   │
                                   ▼
                          ┌─────────────────┐
                          │ tcp_nopush waits │
                          │ to group packets │
                          └─────────────────┘

Build-Up - 6 Steps

1

Foundation

What is sendfile in nginx

Concept: sendfile is a system call that lets nginx send files directly from disk to the network without extra copying.

Normally, when nginx sends a file, it reads the file into memory and then sends it over the network. sendfile skips the memory copy step by telling the operating system to send the file directly from disk to the network socket.

Result

Files are sent faster and with less CPU usage.

Understanding sendfile shows how nginx reduces work by letting the OS handle file transfer efficiently.

2

Foundation

What is tcp_nopush in nginx

Concept: tcp_nopush is a TCP option that controls how data packets are grouped before sending to reduce overhead.

When tcp_nopush is enabled, nginx tells the OS to hold small packets and combine them into larger ones before sending. This reduces the number of packets sent and improves network efficiency.

Result

Network packets are larger and fewer, improving speed and reducing overhead.

Knowing tcp_nopush helps understand how network packet size affects performance.

3

Intermediate

How sendfile and tcp_nopush work together

🤔Before reading on: do you think sendfile and tcp_nopush can be used independently without affecting each other? Commit to your answer.

Concept: sendfile and tcp_nopush complement each other to optimize file sending and packet grouping.

sendfile sends file data directly, but without tcp_nopush, packets might be sent immediately in small chunks. tcp_nopush waits to group these chunks into bigger packets, reducing network overhead and improving throughput.

Result

Using both together maximizes file transfer efficiency and network performance.

Understanding their interaction reveals why enabling both is common for high-performance servers.

4

Intermediate

Configuring sendfile and tcp_nopush in nginx

🤔Before reading on: do you think enabling sendfile alone is enough for best performance? Commit to your answer.

Concept: nginx allows enabling sendfile and tcp_nopush via configuration directives to control file sending behavior.

In nginx config, 'sendfile on;' enables sendfile. 'tcp_nopush on;' enables tcp_nopush on Linux. These are set in the http or server block. Example: sendfile on; tcp_nopush on;

Result

nginx uses OS features to optimize file sending and packet grouping.

Knowing how to configure these options is key to tuning nginx for better performance.

5

Advanced

When tcp_nopush can cause delays

🤔Before reading on: do you think tcp_nopush always improves performance? Commit to your answer.

Concept: tcp_nopush can delay sending packets to group them, which may cause latency in some cases.

If nginx waits too long to fill packets, small responses or interactive content might be delayed. This is a tradeoff between throughput and latency. Sometimes disabling tcp_nopush helps reduce delay for small responses.

Result

Performance tuning requires balancing throughput and latency based on workload.

Understanding tcp_nopush tradeoffs helps avoid performance pitfalls in real applications.

6

Expert

sendfile and tcp_nopush internals on Linux

🤔Before reading on: do you think tcp_nopush is the same as TCP_CORK? Commit to your answer.

Concept: tcp_nopush in nginx maps to TCP_CORK socket option on Linux, controlling packet corking behavior.

Linux TCP_CORK holds back packets until enough data is available or cork is removed. nginx uses TCP_CORK when tcp_nopush is on. sendfile uses zero-copy kernel mechanisms. Together, they minimize CPU and network overhead by controlling when and how packets are sent.

Result

Deep understanding of kernel socket options explains nginx's performance tuning.

Knowing the kernel-level details clarifies why tcp_nopush and sendfile behave as they do and how to troubleshoot issues.

Under the Hood

sendfile uses a zero-copy mechanism where the kernel reads file data directly from disk buffers and sends it to the network socket, avoiding copying data to user space. tcp_nopush sets the TCP_CORK option on the socket, which tells the kernel to hold back sending partial packets until a full packet can be formed or the cork is removed. This reduces the number of small packets sent, improving network efficiency.

Why designed this way?

sendfile was designed to reduce CPU load and memory bandwidth by avoiding unnecessary data copying. tcp_nopush (TCP_CORK) was introduced to optimize packet transmission by grouping data into full-sized packets, reducing overhead and improving throughput. Alternatives like sending data in user space or sending many small packets were less efficient and caused higher CPU and network load.

┌───────────────┐          ┌───────────────┐          ┌───────────────┐
│   Disk Cache  │─────────▶│   Kernel TCP   │─────────▶│ Network Driver │
│ (File Data)   │ zero-copy│   Socket with  │  packets │               │
└───────────────┘  sendfile│   TCP_CORK on │─────────▶│               │
                            └───────────────┘          └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does enabling sendfile always improve performance? Commit yes or no.

Common Belief: Enabling sendfile always makes nginx faster with no downsides.

Reality: sendfile improves performance for static files but can cause issues with dynamic content or when using certain filters that modify output.

Why it matters: Misusing sendfile can cause corrupted responses or broken features, leading to bad user experience.

Quick: Is tcp_nopush the same as disabling Nagle's algorithm? Commit yes or no.

Common Belief: tcp_nopush disables Nagle's algorithm and sends packets immediately.

Reality: tcp_nopush (TCP_CORK) delays sending packets to group data, opposite to Nagle's algorithm which tries to reduce small packets by buffering.

Why it matters: Confusing these can lead to wrong tuning decisions, hurting performance or increasing latency.

Quick: Can tcp_nopush cause delays in sending data? Commit yes or no.

Common Belief: tcp_nopush always speeds up data sending without delay.

Reality: tcp_nopush can delay sending small packets to form larger ones, which may increase latency for some responses.

Why it matters: Ignoring this can cause slow responses in interactive applications.

Quick: Does tcp_nopush work the same on all operating systems? Commit yes or no.

Common Belief: tcp_nopush behaves identically on Linux, BSD, and other OSes.

Reality: tcp_nopush maps to different socket options on different OSes (e.g., TCP_NOPUSH on BSD, TCP_CORK on Linux) with subtle behavior differences.

Why it matters: Assuming identical behavior can cause cross-platform bugs and performance issues.

Expert Zone

1

tcp_nopush must be carefully paired with sendfile; enabling tcp_nopush without sendfile often has no benefit.

2

On Linux, tcp_nopush uses TCP_CORK which can cause head-of-line blocking if not managed properly.

3

Some nginx modules or filters disable sendfile internally, so configuration alone may not enable it fully.

When NOT to use

Avoid sendfile when serving dynamic content that requires output modification or compression. tcp_nopush should be disabled for low-latency interactive applications. Alternatives include using proxy buffering or tuning TCP_NODELAY for small packet sending.

Production Patterns

In production, sendfile and tcp_nopush are enabled together for static file serving to maximize throughput. For APIs or dynamic content, they are often disabled to avoid delays. Monitoring tools track packet sizes and latency to tune these settings per workload.

Connections

Zero-copy networking

sendfile is a classic example of zero-copy networking techniques.

Understanding sendfile helps grasp how zero-copy reduces CPU and memory usage in network file transfers.

TCP socket options

tcp_nopush is a TCP socket option controlling packet sending behavior.

Knowing tcp_nopush deepens understanding of how TCP options affect network performance and latency.

Logistics and delivery optimization

sendfile and tcp_nopush optimize data delivery like logistics optimizes package shipping.

Seeing data transfer as a delivery problem helps appreciate why grouping and direct transfer improve efficiency.

Common Pitfalls

#1 Enabling sendfile without considering dynamic content.

Wrong approach:

sendfile on;
# Used with dynamic content that nginx modifies

Correct approach:

# Disable sendfile for dynamic content
sendfile off;

Root cause:

Misunderstanding that sendfile only works well with static files and can break dynamic responses.

#2 Enabling tcp_nopush without sendfile.

Wrong approach:

tcp_nopush on;
sendfile off;

Correct approach:

sendfile on;
tcp_nopush on;

Root cause:

Not knowing tcp_nopush benefits only apply when sendfile is enabled.

#3 Assuming tcp_nopush reduces latency for all traffic.

Wrong approach:

tcp_nopush on; # for low-latency interactive apps

Correct approach:

tcp_nopush off; # for interactive or small packet traffic

Root cause:

Ignoring that tcp_nopush delays sending small packets to group them.

Key Takeaways

sendfile lets nginx send files directly from disk to network, saving CPU and memory.

tcp_nopush groups small packets into larger ones, improving network efficiency but can add delay.

Using sendfile and tcp_nopush together maximizes static file serving performance.

These options have tradeoffs and should be tuned based on workload type and latency needs.

Understanding kernel socket options behind these settings helps troubleshoot and optimize nginx.