Log rotation script in Bash Scripting - Deep Dive

Overview - Log rotation script

What is it?

A log rotation script is a small program that automatically manages log files by renaming, compressing, or deleting old logs to keep disk space free and logs organized. Logs are files where programs record their activity, errors, or important events. Without rotation, these files grow endlessly, making it hard to find recent information and risking full storage. The script helps keep logs manageable and ensures the system runs smoothly.

Why it matters

Without log rotation, log files can grow very large, filling up disk space and slowing down systems. This can cause crashes or lost data. Log rotation scripts prevent this by regularly cleaning up old logs and keeping only recent ones. This saves storage, improves performance, and helps system administrators find relevant information quickly when troubleshooting.

Where it fits

Before learning log rotation scripts, you should understand basic shell scripting and file management commands in bash. After mastering log rotation, you can explore system monitoring, automated backups, and advanced log analysis tools to maintain healthy systems.

Mental Model

Core Idea

A log rotation script automatically renames, compresses, and deletes old log files to keep logs organized and disk space free.

Think of it like...

It's like cleaning out your mailbox regularly: you keep recent letters handy but archive or throw away old ones so the mailbox doesn't overflow.

┌───────────────┐
│ Current Log   │
│ (app.log)     │
└──────┬────────┘
       │ Rotate (rename, compress)
       ▼
┌───────────────┐
│ Old Logs      │
│ app.log.1.gz  │
│ app.log.2.gz  │
│ ...           │
└──────┬────────┘
       │ Delete oldest if too many
       ▼
┌───────────────┐
│ Disk Space OK │
└───────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding log files basics

Concept: Learn what log files are and why they grow over time.

Log files are text files where programs write messages about their activity. For example, a web server writes every visitor's request to a log file. Over time, these files get bigger because new messages keep adding. If not managed, they can fill up the disk.

Result

You understand that logs grow continuously and need management to avoid disk space problems.

Knowing that logs grow endlessly explains why automatic management is necessary to keep systems healthy.

2

Foundation

Basic bash file operations

Concept: Learn simple bash commands to rename, compress, and delete files.

Commands like mv rename files, gzip compresses files, and rm deletes files. For example: mv app.log app.log.1 gzip app.log.1 rm app.log.5.gz These commands form the building blocks of log rotation.

Result

You can manually rename, compress, and delete files using bash commands.

Mastering these commands is essential because log rotation scripts automate these exact steps.

3

Intermediate

Creating a simple rotation script

🤔Before reading on: do you think a script should rename the current log before compressing it, or compress first? Commit to your answer.

Concept: Write a script that renames the current log file and compresses the renamed file.

#!/bin/bash LOGFILE="app.log" # Rename current log mv "$LOGFILE" "$LOGFILE.1" # Compress the renamed log gzip "$LOGFILE.1" # Create a new empty log file > "$LOGFILE" # Output message echo "Log rotated: $LOGFILE -> $LOGFILE.1.gz"

Result

Running this script renames app.log to app.log.1, compresses it to app.log.1.gz, and creates a new empty app.log.

Understanding the order of operations prevents losing log data and ensures the new log file is ready for fresh entries.

4

Intermediate

Managing multiple rotated logs

🤔Before reading on: should the script delete the oldest log first or rename all logs before deleting? Commit to your answer.

Concept: Extend the script to keep a fixed number of rotated logs and delete the oldest when exceeding the limit.

#!/bin/bash LOGFILE="app.log" MAX=5 # Delete oldest if exists if [ -f "$LOGFILE.$MAX.gz" ]; then rm "$LOGFILE.$MAX.gz" fi # Shift older logs for ((i=MAX-1; i>=1; i--)); do if [ -f "$LOGFILE.$i.gz" ]; then mv "$LOGFILE.$i.gz" "$LOGFILE.$((i+1)).gz" fi done # Rotate current log mv "$LOGFILE" "$LOGFILE.1" gzip "$LOGFILE.1" > "$LOGFILE" echo "Rotated and cleaned logs"

Result

The script keeps only 5 compressed rotated logs, deleting the oldest and shifting others up by one number.

Knowing how to manage multiple logs prevents disk overflow and keeps logs organized by age.

5

Intermediate

Automating rotation with cron

Concept: Learn how to schedule the log rotation script to run automatically at set times using cron.

Cron is a Linux tool to run commands on a schedule. To run the script daily at midnight: 1. Open crontab: crontab -e 2. Add line: 0 0 * * * /path/to/logrotate.sh 3. Save and exit This runs the script every day at midnight without manual intervention.

Result

Log rotation happens automatically every day, keeping logs managed without user action.

Automating log rotation ensures consistent maintenance and reduces human error or forgetfulness.

6

Advanced

Handling log rotation safely

🤔Before reading on: do you think truncating the log file is safer than renaming it? Commit to your answer.

Concept: Learn how to rotate logs without stopping the program writing to them, using truncation instead of renaming.

#!/bin/bash LOGFILE="app.log" # Copy current log to rotated file cp "$LOGFILE" "$LOGFILE.$(date +%Y%m%d)" # Truncate original log without deleting : > "$LOGFILE" # Compress rotated log gzip "$LOGFILE.$(date +%Y%m%d)" echo "Safe rotation done"

Result

The program continues writing to the same log file, which is emptied safely while the old content is saved and compressed.

Understanding file descriptors and truncation avoids losing logs or stopping services during rotation.

7

Expert

Integrating with system logrotate utility

🤔Before reading on: do you think logrotate utilities are always better than custom scripts? Commit to your answer.

Concept: Explore how system tools like logrotate work and when to use them instead of custom scripts.

Linux systems often have a logrotate tool that handles rotation with configuration files. It supports compression, retention, and safe rotation with signals to programs. Example config: /path/to/app.log { daily rotate 7 compress delaycompress missingok notifempty copytruncate } Using logrotate reduces errors and adds features but custom scripts offer flexibility for special cases.

Result

You can choose between custom scripts for simple needs or logrotate for robust, tested rotation management.

Knowing system tools exist helps avoid reinventing the wheel and leverages tested solutions for reliability.

Under the Hood

Log rotation scripts work by manipulating files at the operating system level. When a log file is renamed or moved, the program writing to it may still hold an open file handle pointing to the old file location. This means the program can continue writing to the old file even if renamed, causing confusion. To avoid this, scripts either signal the program to reopen logs or truncate the file in place. Compression reduces disk space by encoding the log data efficiently. Deleting old logs frees space and keeps the system tidy.

Why designed this way?

Log rotation was designed to solve the problem of ever-growing log files that consume disk space and become unwieldy. Early systems lacked automatic rotation, causing crashes and lost logs. The design balances safety (not losing logs), automation (no manual cleanup), and efficiency (compressing old logs). Alternatives like manual deletion were error-prone, and always stopping programs to rotate logs was disruptive, so truncation and signaling became standard.

┌───────────────┐
│ Program writes │
│ to app.log    │
└──────┬────────┘
       │
       │ Rename or copy
       ▼
┌───────────────┐
│ app.log.1.gz  │
└──────┬────────┘
       │
       │ Program still writes
       │ to old file handle
       ▼
┌───────────────┐
│ Truncate or   │
│ signal program│
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ app.log empty │
│ ready for new │
│ logs          │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does renaming a log file immediately stop the program from writing to it? Commit to yes or no.

Common Belief: Renaming a log file stops the program from writing to the old file immediately.

Reality: Programs keep writing to the old file descriptor until they reopen the log, so renaming alone doesn't stop writes.

Why it matters: Assuming renaming stops writes can cause logs to grow unexpectedly or data to be lost if the program overwrites files.

Quick: Is compressing logs before rotating them better than after? Commit to your answer.

Common Belief: Compressing logs before rotating saves more space and is the best practice.

Reality: Logs must be rotated (renamed or copied) before compression; compressing the active log file can corrupt it or lose data.

Why it matters: Compressing active logs risks losing current log entries and breaks log continuity.

Quick: Should you always delete all old logs during rotation? Commit to yes or no.

Common Belief: Deleting all old logs during rotation is best to save disk space.

Reality: Keeping a limited number of old logs is important for troubleshooting and auditing; deleting all logs removes valuable history.

Why it matters: Deleting all logs can make it impossible to investigate past issues or meet compliance requirements.

Quick: Is a custom log rotation script always better than system tools? Commit to yes or no.

Common Belief: Custom scripts are always better because they are tailored to specific needs.

Reality: System tools like logrotate are more reliable, tested, and feature-rich for most cases; custom scripts add complexity and risk.

Why it matters: Overusing custom scripts can cause bugs, maintenance overhead, and missed features that system tools provide.

Expert Zone

1

Log rotation must consider file descriptors: programs may keep writing to old files unless signaled or logs truncated.

2

Compression timing affects performance; delaying compression can reduce CPU load during peak hours.

3

Rotation scripts should handle edge cases like missing logs, empty files, or permission errors gracefully to avoid failures.

When NOT to use

Custom log rotation scripts are not ideal for complex systems with many logs or where reliability is critical; use system tools like logrotate or centralized logging solutions instead.

Production Patterns

In production, log rotation is often combined with centralized log management (e.g., ELK stack) and monitoring alerts to automate troubleshooting and maintain system health.

Connections

File Descriptors in Operating Systems

Log rotation scripts must manage file descriptors correctly to avoid data loss or corruption.

Understanding how programs keep file handles open explains why simple renaming isn't enough for safe log rotation.

Cron Scheduling

Log rotation scripts are often scheduled with cron to automate maintenance tasks.

Knowing cron helps automate repetitive tasks like log rotation, freeing administrators from manual work.

Waste Management Systems

Both log rotation and waste management involve regularly removing or compressing old material to keep the system clean and efficient.

Seeing log rotation as a form of digital waste management helps appreciate its role in system hygiene and resource conservation.

Common Pitfalls

#1 Renaming the log file without creating a new empty log causes the program to continue writing to the old file name, losing new logs.

Wrong approach:

mv app.log app.log.1
gzip app.log.1
# No new app.log created

Correct approach:

mv app.log app.log.1
gzip app.log.1
> app.log # Create new empty log file

Root cause:

Not creating a new log file leaves the original log missing, but the program still writes to the old file handle.

#2 Deleting old logs without limit causes loss of important historical data needed for troubleshooting.

Wrong approach:

rm app.log.*.gz # Deletes all rotated logs every time

Correct approach:

# Keep last 5 logs only
rm app.log.6.gz
mv app.log.5.gz app.log.6.gz
... (shift others)

Root cause:

Not managing retention policy leads to accidental deletion of useful logs.

#3 Compressing the active log file instead of the rotated one corrupts the log and stops logging.

Wrong approach:

gzip app.log # Compress active log file

Correct approach:

mv app.log app.log.1
gzip app.log.1
> app.log

Root cause:

Compressing the active log file breaks the file handle the program writes to.

Key Takeaways

Log rotation scripts keep log files manageable by renaming, compressing, and deleting old logs automatically.

Proper rotation order and handling of file descriptors prevent data loss and ensure continuous logging.

Automating rotation with tools like cron avoids manual errors and keeps systems healthy.

System utilities like logrotate offer robust, tested solutions that often outperform custom scripts.

Understanding the underlying file and process behavior is key to writing safe and effective log rotation scripts.