Logging strategies in HLD - Scalability & System Analysis

Scalability Analysis - Logging strategies

Growth Table: Logging Strategies at Different Scales

Users / Events	100 Users	10K Users	1M Users	100M Users
Log Volume	Low, simple file logs	Moderate, centralized logging	High, distributed log storage	Very high, scalable log pipelines
Storage	Local disk	Centralized server or cloud storage	Distributed storage clusters	Sharded, tiered storage with archiving
Log Processing	Manual or basic scripts	Automated parsing and indexing	Stream processing and alerting	Real-time analytics and AI-based anomaly detection
Latency	Not critical	Near real-time	Real-time or near real-time	Real-time with high throughput
Retention	Short term (days)	Weeks to months	Months to years	Years with cold storage

First Bottleneck

At small scale, the local disk write speed and log file size limits the logging system. As users grow to 10K and beyond, the centralized log storage and processing become bottlenecks due to high write throughput and storage needs. At 1M+ users, the network bandwidth and distributed storage coordination become the main bottlenecks.

Scaling Solutions

• Small scale: Use local file logging with rotation to avoid disk full issues.
• Medium scale: Centralize logs using a log collector (e.g., Fluentd, Logstash) and store in a scalable system like Elasticsearch.
• Large scale: Use distributed log pipelines with Kafka or similar message queues to decouple producers and consumers.
• Very large scale: Implement sharding of logs by source or time, tiered storage (hot/warm/cold), and use cloud storage for archival.
• Across scales: Use sampling, log level filtering, and compression to reduce volume.

Back-of-Envelope Cost Analysis

Assuming 1 log event is ~1 KB:

• 100 users generating 10 logs/sec = 1,000 logs/sec = ~1 MB/s storage and bandwidth.
• 10K users generating 10 logs/sec = 100,000 logs/sec = ~100 MB/s bandwidth and storage.
• 1M users generating 10 logs/sec = 10M logs/sec = ~10 GB/s bandwidth and storage.
• 100M users generating 10 logs/sec = 1B logs/sec = ~1 TB/s bandwidth and storage (requires massive distributed systems).

Storage needs grow quickly; retention policies and compression are critical to control costs.

Interview Tip

When discussing logging scalability, start by clarifying log volume and retention needs. Then identify bottlenecks at each scale. Propose incremental solutions like centralization, message queues, and distributed storage. Mention cost trade-offs and monitoring for failures.

Self Check

Your database handles 1000 QPS for logging writes. Traffic grows 10x to 10,000 QPS. What do you do first?

Answer: Introduce a message queue (e.g., Kafka) to buffer and decouple log writes from the database. This prevents overload and allows asynchronous processing. Also consider adding read replicas or scaling the database vertically/horizontally.

Key Result

Logging systems start simple with local files but must evolve to distributed pipelines and storage as user and log volume grow to millions, with bottlenecks shifting from disk IO to network and storage coordination.

Practice

1. What is the main purpose of using different log levels like INFO, WARN, and ERROR in logging strategies?

easy

A. To categorize logs by importance and help filter relevant information

B. To make logs look colorful and attractive

C. To increase the size of log files for better storage

D. To encrypt log messages for security

Solution

1. Step 1: Understand log levels

   Log levels like INFO, WARN, and ERROR help mark how important or severe a log message is.

2. Step 2: Purpose of categorizing logs

   This categorization helps users quickly find important messages and ignore less critical ones.

3. Final Answer:

   To categorize logs by importance and help filter relevant information -> Option A

4. Quick Check:

   Log levels = Categorize importance [OK]

Hint: Log levels help sort messages by importance fast [OK]

Common Mistakes:

• Thinking log levels are for decoration
• Assuming log levels increase file size intentionally
• Confusing log levels with encryption

2. Which of the following is the correct syntax to configure a log message format in a typical logging configuration file?

easy

A. format: asctime, levelname, message

B. format = "%(asctime)s - %(levelname)s - %(message)s"

C. format -> asctime | levelname | message

D. format = asctime + levelname + message

Solution

1. Step 1: Recognize standard log format syntax

   Typical logging uses placeholders like %(asctime)s for timestamp, %(levelname)s for log level, and %(message)s for the log text.

2. Step 2: Match correct syntax

   format = "%(asctime)s - %(levelname)s - %(message)s" uses the correct placeholder format with percent signs and parentheses, matching common logging config files.

3. Final Answer:

   format = "%(asctime)s - %(levelname)s - %(message)s" -> Option B

4. Quick Check:

   Correct placeholders = format = "%(asctime)s - %(levelname)s - %(message)s" [OK]

Hint: Look for %()s placeholders in format strings [OK]

Common Mistakes:

• Using commas or arrows instead of placeholders
• Missing percent signs and parentheses
• Concatenating strings without placeholders

3. Given this log configuration snippet:

log_level = "WARN"
log_destination = "file"

What will happen if the system logs an INFO message?

medium

A. The INFO message will be saved to the file

B. The system will crash due to invalid log level

C. The INFO message will be printed to the console only

D. The INFO message will be ignored and not saved

Solution

1. Step 1: Understand log level filtering

   Setting log_level to WARN means only WARN and ERROR messages (higher importance) are saved; INFO is lower priority.

2. Step 2: Effect on INFO messages

   INFO messages are below WARN level, so they will be ignored and not saved to the file.

3. Final Answer:

   The INFO message will be ignored and not saved -> Option D

4. Quick Check:

   Log level filter blocks lower priority logs [OK]

Hint: Logs below set level are ignored [OK]

Common Mistakes:

• Assuming all logs are saved regardless of level
• Thinking INFO logs print to console automatically
• Believing invalid log level causes crash

4. You have a logging setup that writes logs to a file, but the file remains empty even after errors occur. Which is the most likely cause?

medium

A. The log level is set too high, filtering out error messages

B. The logging format string is missing placeholders

C. The log file path is incorrect or lacks write permission

D. The system does not support file logging

Solution

1. Step 1: Check log level filtering

   Error messages are high priority, so a high log level would not filter them out; this is unlikely the cause.

2. Step 2: Verify file path and permissions

   If the file path is wrong or the system user cannot write to the file, logs won't be saved, causing an empty file.

3. Final Answer:

   The log file path is incorrect or lacks write permission -> Option C

4. Quick Check:

   File permission issues cause empty logs [OK]

Hint: Check file path and permissions first [OK]

Common Mistakes:

• Blaming log level for missing error logs
• Ignoring file system permissions
• Assuming format string affects file writing

5. You want to design a logging strategy for a web app that must:
- Capture all errors and warnings
- Save detailed info logs only during debugging
- Send critical errors to an external alert system

Which combination of strategies best meets these needs?

hard

A. Use log level WARN for normal runs, switch to INFO for debugging, and configure alerts for ERROR logs

B. Always use log level INFO and save all logs to a single file without alerts

C. Use log level ERROR only and ignore INFO and WARN logs

D. Send all logs to the alert system regardless of level

Solution

1. Step 1: Match log levels to requirements

   WARN level captures warnings and errors; INFO level includes detailed info logs for debugging.

2. Step 2: Configure alerting for critical errors

   Setting alerts on ERROR logs ensures critical issues trigger notifications externally.

3. Step 3: Evaluate options

   Use log level WARN for normal runs, switch to INFO for debugging, and configure alerts for ERROR logs matches all requirements: dynamic log level and alerting. Others miss one or more needs.

4. Final Answer:

   Use log level WARN for normal runs, switch to INFO for debugging, and configure alerts for ERROR logs -> Option A

5. Quick Check:

   Dynamic levels + alerts = Use log level WARN for normal runs, switch to INFO for debugging, and configure alerts for ERROR logs [OK]

Hint: Adjust log level by mode and alert on errors [OK]

Common Mistakes:

• Logging everything all the time wastes resources
• Ignoring alerting for critical errors
• Using too high log level and missing info logs