System logs (/var/log) in Linux CLI - Deep Dive

Overview - System logs (/var/log)

What is it?

System logs are files that record events and messages generated by the operating system and applications. These logs are stored in the /var/log directory on Linux systems. They help track what the system is doing, record errors, and provide information for troubleshooting. Anyone can read these logs to understand system behavior or diagnose problems.

Why it matters

Without system logs, it would be very hard to know what happened on a computer when something goes wrong. Imagine a car without a dashboard or history of trips; you wouldn't know if the engine had problems or when. System logs provide a detailed history that helps fix issues quickly and keep the system safe and stable.

Where it fits

Before learning about system logs, you should understand basic Linux commands and file system structure. After mastering logs, you can learn about automated monitoring, alerting tools, and log analysis techniques to maintain systems efficiently.

Mental Model

Core Idea

System logs are like a diary where the computer writes down everything important it does or notices, so you can read it later to understand or fix problems.

Think of it like...

Think of system logs as a security camera recording everything happening in a building. If something breaks or goes missing, you review the footage to see what happened and when.

┌───────────────┐
│ /var/log      │
├───────────────┤
│ syslog       │ ← General system messages
│ auth.log     │ ← Security and login events
│ kern.log     │ ← Kernel messages
│ dmesg        │ ← Boot and hardware info
│ apache2/     │ ← Web server logs
│ ...          │
└───────────────┘

Build-Up - 7 Steps

1

Foundation

What Are System Logs

Concept: Introduce the idea of logs as files that record system events.

System logs are text files stored in /var/log. They keep records of what the system and applications do. For example, when you log in, the system writes a message in a log file. These files grow over time and help track system health.

Result

You understand that logs are files you can open and read to see system activity.

Knowing that logs are simple files demystifies them and shows you can explore system history anytime.

2

Foundation

Common Log Files in /var/log

Concept: Learn about the main log files and what they record.

Some important logs are: - syslog: general system messages - auth.log: login and security events - kern.log: kernel messages - dmesg: boot and hardware info You can open these files with commands like 'cat' or 'less'.

Result

You can identify key logs and know what kind of information each holds.

Recognizing log types helps you quickly find the right information when troubleshooting.

3

Intermediate

Reading Logs with Command-Line Tools

🤔Before reading on: do you think 'tail -f' shows the entire log or only new entries? Commit to your answer.

Concept: Use commands to view and search logs efficiently.

Commands: - cat /var/log/syslog: shows whole file - less /var/log/auth.log: scroll through file - tail -n 20 /var/log/kern.log: last 20 lines - tail -f /var/log/syslog: live updates as new lines appear - grep 'error' /var/log/syslog: find lines with 'error' These help you find and monitor important events.

Result

You can view logs in real-time and search for specific messages quickly.

Mastering these commands saves time and helps catch issues as they happen.

4

Intermediate

Log Rotation and Management

🤔Before reading on: do you think logs grow forever or get managed automatically? Commit to your answer.

Concept: Understand how logs are kept from growing too large using rotation.

Logs can become very large, so Linux uses 'logrotate' to archive old logs and start fresh ones. For example, syslog might be saved as syslog.1, syslog.2.gz, etc. This keeps disk space under control and organizes logs by date.

Result

You know logs are automatically archived and compressed to save space.

Understanding log rotation prevents confusion when old logs disappear or are compressed.

5

Intermediate

Permissions and Security of Logs

Concept: Learn who can read or write logs and why it matters.

Log files often contain sensitive info like login attempts. Usually, only root or admin users can read them. For example, 'ls -l /var/log/auth.log' shows permissions. Protecting logs prevents attackers from hiding their tracks or stealing info.

Result

You understand the importance of log file permissions for system security.

Knowing permissions helps you keep logs secure and trustworthy.

6

Advanced

Automating Log Monitoring with Scripts

🤔Before reading on: do you think scripts can watch logs and alert you automatically? Commit to your answer.

Concept: Use simple scripts to watch logs and notify on important events.

You can write shell scripts that use 'tail -f' and 'grep' to watch logs for keywords like 'error' or 'failed'. When found, the script can send an email or message. This helps catch problems early without manual checking.

Result

You can automate log watching to get alerts on critical issues.

Automating log checks saves time and improves system reliability by catching issues fast.

7

Expert

Understanding Systemd Journal vs /var/log

🤔Before reading on: do you think systemd journal replaces /var/log files or works alongside? Commit to your answer.

Concept: Explore how modern Linux systems use systemd journal for logs and how it relates to /var/log.

Systemd journal collects logs in a binary format, accessible with 'journalctl'. It can store logs in memory or disk and offers advanced filtering. Some systems forward journal logs to /var/log files for compatibility. Understanding both helps manage logs effectively.

Result

You know the difference and relationship between systemd journal and traditional /var/log files.

Knowing this prevents confusion when logs appear in different places and helps use the best tool for log analysis.

Under the Hood

When events happen, system components write messages to log files or the systemd journal. These messages include timestamps, source info, and event details. The kernel, system services, and applications use logging APIs to send messages. Logrotate runs periodically to archive and compress old logs, preventing disk overflow.

Why designed this way?

Logs are stored as text files for simplicity and compatibility with many tools. Systemd journal was introduced to provide structured, indexed logs for faster searching and richer metadata. Log rotation was designed to manage disk space automatically, avoiding manual cleanup.

┌───────────────┐
│ System Events │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Logging APIs  │
└──────┬────────┘
       │
       ▼
┌───────────────┐          ┌───────────────┐
│ /var/log Files│◄─────────┤ logrotate     │
└───────────────┘          └───────────────┘
       │
       ▼
┌───────────────┐
│ systemd journal│
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do you think all system logs are stored only in /var/log? Commit to yes or no.

Common Belief: All system logs are always stored as text files in /var/log.

Reality: Modern Linux systems also use systemd journal, which stores logs in a binary format outside /var/log.

Why it matters: Assuming logs are only in /var/log can cause you to miss important messages or waste time searching in the wrong place.

Quick: Do you think you can edit log files directly to fix errors? Commit to yes or no.

Common Belief: You can safely edit log files to remove or fix entries.

Reality: Editing logs is discouraged because it breaks the integrity and audit trail of system events.

Why it matters: Modifying logs can hide problems or cause security issues, making troubleshooting and audits unreliable.

Quick: Do you think log rotation deletes old logs immediately? Commit to yes or no.

Common Belief: Log rotation deletes old logs permanently as soon as it runs.

Reality: Log rotation archives and compresses old logs, keeping them for a configurable time before deletion.

Why it matters: Believing logs are deleted immediately can cause you to miss historical data needed for long-term troubleshooting.

Quick: Do you think anyone can read all log files by default? Commit to yes or no.

Common Belief: All users can read all log files by default.

Reality: Many logs have restricted permissions to protect sensitive information; only admins or root can read them.

Why it matters: Assuming open access can lead to security risks or confusion when users cannot access needed logs.

Expert Zone

1

Systemd journal supports structured logging with fields, enabling powerful filtering beyond simple text search.

2

Log rotation timing and compression settings can be tuned per log file to balance performance and storage.

3

Some applications write logs directly to custom locations, so /var/log is not the only place to check.

When NOT to use

Relying solely on manual log inspection is inefficient for large or complex systems; use centralized log management tools like ELK stack or Splunk instead.

Production Patterns

In production, logs are often forwarded to remote servers for aggregation and analysis. Automated alerts trigger on specific log patterns to notify admins immediately.

Connections

Event-Driven Programming

System logs record events that trigger actions, similar to how event-driven programs respond to events.

Understanding logs as event records helps grasp how systems react to changes and errors in real time.

Forensic Science

Both system logs and forensic evidence provide a timeline of events to reconstruct what happened.

Knowing how logs serve as digital evidence highlights their importance in security and troubleshooting.

Database Indexing

Systemd journal indexes log entries for fast searching, similar to how databases index data.

Recognizing indexing in logs explains why some tools are faster and more efficient for log analysis.

Common Pitfalls

#1 Trying to read large log files with 'cat' causing screen flooding.

Wrong approach:

cat /var/log/syslog

Correct approach:

less /var/log/syslog

Root cause:

Not knowing that 'cat' outputs the entire file at once, overwhelming the terminal.

#2 Ignoring log rotation causing disk space to fill up.

Wrong approach:

Never checking or configuring logrotate, letting logs grow indefinitely.

Correct approach:

Configure and verify logrotate settings to archive and compress logs regularly.

Root cause:

Lack of awareness about log management leads to system crashes or slowdowns.

#3 Running log viewing commands without root when logs are restricted.

Wrong approach:

less /var/log/auth.log (as normal user)

Correct approach:

sudo less /var/log/auth.log

Root cause:

Not understanding file permissions prevents access to important logs.

Key Takeaways

System logs in /var/log are essential records of system and application events that help diagnose and fix problems.

Logs are plain text files but can be large and complex; using command-line tools efficiently is key to managing them.

Log rotation automatically archives old logs to save disk space and keep logs organized over time.

Modern Linux systems also use systemd journal, a binary log system with advanced features alongside traditional logs.

Proper permissions protect logs from unauthorized access, maintaining system security and trustworthiness.