Log rotation in Nginx - Deep Dive

Overview - Log rotation

What is it?

Log rotation is the process of managing log files by periodically archiving old logs and creating new ones. This keeps log files from growing too large and consuming too much disk space. In nginx, logs record server activity like requests and errors, which help monitor and troubleshoot the server. Without log rotation, logs can become huge and hard to handle.

Why it matters

Without log rotation, log files grow endlessly, filling up disk space and slowing down the server. This can cause crashes or lost data. Rotating logs keeps the system healthy and makes it easier to find recent events. It also helps with compliance and auditing by organizing logs into manageable chunks.

Where it fits

Before learning log rotation, you should understand basic nginx logging and file system concepts. After mastering log rotation, you can explore log analysis tools and monitoring systems that use these rotated logs for insights.

Mental Model

Core Idea

Log rotation is like regularly packing away old papers into labeled boxes so your desk stays clean and you can find recent notes easily.

Think of it like...

Imagine your desk is full of papers (logs). If you never clear them, the desk becomes cluttered and you can't work well. By moving old papers into boxes (rotated logs), your desk stays tidy and you can focus on current work.

┌───────────────┐
│ Current Log   │  <-- Active log file nginx writes to
├───────────────┤
│ Old Logs      │  <-- Archived logs, compressed and timestamped
│ (rotated)     │
└───────────────┘

Build-Up - 6 Steps

1

Foundation

Understanding nginx log files

Concept: Learn what nginx log files are and what they contain.

Nginx creates two main log files: access.log records every request to the server, and error.log records problems nginx encounters. These files grow as the server runs, storing text lines with details like IP addresses, timestamps, and error messages.

Result

You know where nginx stores logs and what kind of information they hold.

Understanding what logs contain helps you see why managing their size is important.

2

Foundation

Why logs need rotation

Concept: Recognize the problems caused by unlimited log growth.

If logs grow without limit, they consume disk space and slow down reading or searching. Large logs can cause the server to run out of space, leading to failures or lost logs. Rotating logs means moving old logs aside and starting fresh ones.

Result

You understand the risks of not rotating logs and the need for a solution.

Knowing the risks motivates the use of log rotation to keep systems stable.

3

Intermediate

Using logrotate tool with nginx

🤔Before reading on: do you think logrotate changes nginx configuration or just manages files? Commit to your answer.

Concept: Learn how the external tool logrotate manages nginx logs without changing nginx itself.

Logrotate is a Linux tool that moves old log files and creates new ones based on size or time. It can compress old logs and delete very old ones. For nginx, logrotate is configured to rotate access.log and error.log regularly, usually daily or weekly, without stopping nginx.

Result

You can set up logrotate to keep nginx logs manageable and compressed automatically.

Understanding that logrotate works outside nginx helps avoid confusion about how logs are rotated.

4

Intermediate

Configuring nginx for smooth rotation

🤔Before reading on: do you think nginx needs to be restarted to rotate logs? Commit to your answer.

Concept: Learn how nginx handles log files during rotation and how to signal it to reopen logs.

Nginx keeps log files open while running. After logrotate moves the old log, nginx must be told to close and reopen logs to write to the new file. This is done by sending nginx a 'USR1' signal, which tells it to reopen logs without restarting the server.

Result

You know how to configure logrotate to signal nginx properly, avoiding downtime.

Knowing how nginx handles open files prevents log loss or server restarts during rotation.

5

Advanced

Customizing logrotate for nginx needs

🤔Before reading on: do you think rotating logs by size or by time is better? Commit to your answer.

Concept: Explore options to rotate logs based on size, time, compression, and retention policies.

Logrotate can rotate logs daily, weekly, or when they reach a certain size. You can compress old logs to save space and keep only a set number of old logs before deleting them. These settings help balance disk use and log availability. For nginx, daily rotation with compression and keeping 7 days is common.

Result

You can tailor log rotation to your server's traffic and storage limits.

Understanding rotation options helps optimize log management for different workloads.

6

Expert

Handling log rotation in high-traffic environments

🤔Before reading on: do you think standard logrotate is enough for very busy nginx servers? Commit to your answer.

Concept: Learn challenges and solutions for rotating logs on busy servers without losing data or performance.

High-traffic servers generate logs rapidly, making rotation tricky. Standard logrotate may lag or miss logs if rotation is slow. Solutions include using nginx's built-in log rotation with the 'copytruncate' option, or using advanced logging systems like syslog or external log aggregators. Also, asynchronous signaling and careful timing prevent log loss.

Result

You understand advanced strategies to keep logs safe and performance high under heavy load.

Knowing these challenges prepares you to design robust logging for production-grade nginx setups.

Under the Hood

Nginx writes logs by keeping file descriptors open to log files. When logrotate moves or renames these files, nginx still writes to the old file descriptor unless told otherwise. Sending the USR1 signal tells nginx to close and reopen log files, so it writes to the new files. Logrotate runs scripts to rename, compress, and delete logs based on rules.

Why designed this way?

This design avoids stopping nginx during rotation, which would cause downtime. Using signals is lightweight and fast. External tools like logrotate separate concerns: nginx focuses on serving, while logrotate manages files. Alternatives like restarting nginx were rejected due to service disruption.

┌───────────────┐       rotate logs       ┌───────────────┐
│ nginx writes  │ ─────────────────────> │ logrotate     │
│ to access.log │                       │ renames logs  │
└──────┬────────┘                       └──────┬────────┘
       │ send USR1 signal                     │
       │                                     │
       ▼                                     ▼
┌───────────────┐                     ┌───────────────┐
│ nginx closes  │                     │ compressed    │
│ and reopens   │                     │ old logs      │
│ log files     │                     └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does nginx automatically rotate logs by itself? Commit to yes or no.

Common Belief: Nginx automatically rotates its logs without extra tools or configuration.

Reality: Nginx does not rotate logs by itself; it continuously writes to the same files until external rotation happens.

Why it matters: Believing nginx rotates logs can cause disk space to fill unexpectedly, leading to server crashes.

Quick: Is restarting nginx required after every log rotation? Commit to yes or no.

Common Belief: You must restart nginx after rotating logs to make it write to new files.

Reality: You only need to send a USR1 signal to nginx to reopen logs; restarting is unnecessary and causes downtime.

Why it matters: Restarting nginx unnecessarily causes service interruptions and reduces uptime.

Quick: Does logrotate delete logs immediately after rotation? Commit to yes or no.

Common Belief: Logrotate deletes old logs right after rotating them.

Reality: Logrotate usually keeps a set number of old logs and compresses them before deletion based on configuration.

Why it matters: Misunderstanding retention can lead to losing important logs or wasting disk space.

Quick: Is logrotate always reliable on very busy servers? Commit to yes or no.

Common Belief: Logrotate works perfectly on all servers regardless of traffic.

Reality: On very busy servers, logrotate can lag or miss logs; special configurations or tools are needed.

Why it matters: Ignoring this can cause log loss or performance issues in production.

Expert Zone

1

Nginx’s USR1 signal only reopens log files; it does not rotate them, so external tools must handle file renaming and compression.

2

Using 'copytruncate' in logrotate copies the log and truncates the original file, avoiding the need to signal nginx but risking log loss during copy.

3

Log rotation timing should consider peak traffic to avoid rotating logs during heavy load, which can cause delays or missed entries.

When NOT to use

Standard logrotate is not ideal for extremely high-traffic or distributed nginx setups. Instead, use centralized logging systems like Fluentd, Logstash, or cloud logging services that collect logs in real time and handle rotation and storage externally.

Production Patterns

In production, nginx logs are often rotated daily with compression and retention of 7-30 days. Signals to nginx are automated in logrotate scripts. For large systems, logs are shipped to centralized servers or cloud platforms for analysis, reducing local disk use and improving monitoring.

Connections

Systemd Journal

Alternative logging system

Understanding log rotation helps appreciate how systemd journal manages logs differently by centralizing and rotating logs automatically.

Database Archiving

Similar data lifecycle management

Both log rotation and database archiving organize growing data into manageable chunks to maintain performance and storage.

Library Book Lending

Data lifecycle and retention analogy

Like rotating logs, libraries move old books to storage to keep shelves manageable, balancing access and space.

Common Pitfalls

#1 Not signaling nginx after log rotation

Wrong approach:

logrotate script rotates logs but does not send 'kill -USR1' to nginx

Correct approach:

logrotate script rotates logs and runs 'kill -USR1 $(pidof nginx)' to reopen logs

Root cause:

Misunderstanding that nginx keeps old log files open and needs a signal to switch to new files

#2 Using 'copytruncate' without understanding risks

Wrong approach:

logrotate uses 'copytruncate' to rotate nginx logs on a busy server

Correct approach:

Use rename rotation with USR1 signal on busy servers to avoid losing log entries

Root cause:

Not realizing 'copytruncate' can lose logs during the copy phase under heavy write load

#3 Rotating logs too frequently or during peak traffic

Wrong approach:

Scheduling logrotate to run every minute on a high-traffic nginx server

Correct approach:

Schedule logrotate during low traffic periods, like nightly, to reduce performance impact

Root cause:

Ignoring server load patterns and the cost of rotation operations

Key Takeaways

Log rotation prevents nginx log files from growing too large and consuming all disk space.

Nginx does not rotate logs by itself; external tools like logrotate handle this task.

After rotating logs, nginx must be signaled to reopen log files to continue logging correctly.

Proper configuration of rotation frequency, compression, and retention balances disk use and log availability.

High-traffic servers require special care in log rotation to avoid data loss and performance issues.