Logging strategies in HLD - Architecture Diagram

System Overview - Logging strategies

This system shows how logging is handled in a scalable application. Logs capture important events and errors from services. The system ensures logs are collected, stored, and analyzed efficiently without slowing down the main application.

Architecture Diagram

User
  |
  v
Load Balancer
  |
  v
Application Service
  |
  +--------------------+
  |                    |
  v                    v
Log Collector       Main Database
  |
  v
Log Storage
  |
  v
Log Analysis Service

Components

User

user

End user interacting with the application

Load Balancer

load_balancer

Distributes incoming requests evenly to application services

Application Service

service

Handles user requests and generates logs for events and errors

Log Collector

service

Receives logs asynchronously from application services to avoid slowing them down

Main Database

database

Stores application data, separate from logs

Log Storage

database

Stores collected logs in a scalable and searchable format

Log Analysis Service

service

Processes and analyzes logs for monitoring and alerting

Request Flow - 6 Hops

User → Load Balancer

Load Balancer → Application Service

Application Service → Main Database

Application Service → Log Collector

Log Collector → Log Storage

Log Analysis Service → Log Storage

Failure Scenario

Component Fails:Log Collector

Impact:Logs from application services are lost or delayed, reducing monitoring visibility but user requests still succeed

Mitigation:Use retry mechanisms and buffer logs locally in application services until collector is available again

Architecture Quiz - 3 Questions

Why does the Application Service send logs asynchronously to the Log Collector?

A To avoid slowing down user request processing

B To ensure logs are stored before responding to users

C To reduce network traffic

D To encrypt logs before sending

Design Principle

This design separates logging from main request processing to keep the application fast and reliable. Asynchronous log collection and dedicated storage allow scalable log management without impacting user experience.

Practice

1. What is the main purpose of using different log levels like INFO, WARN, and ERROR in logging strategies?

easy

A. To categorize logs by importance and help filter relevant information

B. To make logs look colorful and attractive

C. To increase the size of log files for better storage

D. To encrypt log messages for security

Solution

1. Step 1: Understand log levels

   Log levels like INFO, WARN, and ERROR help mark how important or severe a log message is.

2. Step 2: Purpose of categorizing logs

   This categorization helps users quickly find important messages and ignore less critical ones.

3. Final Answer:

   To categorize logs by importance and help filter relevant information -> Option A

4. Quick Check:

   Log levels = Categorize importance [OK]

Hint: Log levels help sort messages by importance fast [OK]

Common Mistakes:

• Thinking log levels are for decoration
• Assuming log levels increase file size intentionally
• Confusing log levels with encryption

2. Which of the following is the correct syntax to configure a log message format in a typical logging configuration file?

easy

A. format: asctime, levelname, message

B. format = "%(asctime)s - %(levelname)s - %(message)s"

C. format -> asctime | levelname | message

D. format = asctime + levelname + message

Solution

1. Step 1: Recognize standard log format syntax

   Typical logging uses placeholders like %(asctime)s for timestamp, %(levelname)s for log level, and %(message)s for the log text.

2. Step 2: Match correct syntax

   format = "%(asctime)s - %(levelname)s - %(message)s" uses the correct placeholder format with percent signs and parentheses, matching common logging config files.

3. Final Answer:

   format = "%(asctime)s - %(levelname)s - %(message)s" -> Option B

4. Quick Check:

   Correct placeholders = format = "%(asctime)s - %(levelname)s - %(message)s" [OK]

Hint: Look for %()s placeholders in format strings [OK]

Common Mistakes:

• Using commas or arrows instead of placeholders
• Missing percent signs and parentheses
• Concatenating strings without placeholders

3. Given this log configuration snippet:

log_level = "WARN"
log_destination = "file"

What will happen if the system logs an INFO message?

medium

A. The INFO message will be saved to the file

B. The system will crash due to invalid log level

C. The INFO message will be printed to the console only

D. The INFO message will be ignored and not saved

Solution

1. Step 1: Understand log level filtering

   Setting log_level to WARN means only WARN and ERROR messages (higher importance) are saved; INFO is lower priority.

2. Step 2: Effect on INFO messages

   INFO messages are below WARN level, so they will be ignored and not saved to the file.

3. Final Answer:

   The INFO message will be ignored and not saved -> Option D

4. Quick Check:

   Log level filter blocks lower priority logs [OK]

Hint: Logs below set level are ignored [OK]

Common Mistakes:

• Assuming all logs are saved regardless of level
• Thinking INFO logs print to console automatically
• Believing invalid log level causes crash

4. You have a logging setup that writes logs to a file, but the file remains empty even after errors occur. Which is the most likely cause?

medium

A. The log level is set too high, filtering out error messages

B. The logging format string is missing placeholders

C. The log file path is incorrect or lacks write permission

D. The system does not support file logging

Solution

1. Step 1: Check log level filtering

   Error messages are high priority, so a high log level would not filter them out; this is unlikely the cause.

2. Step 2: Verify file path and permissions

   If the file path is wrong or the system user cannot write to the file, logs won't be saved, causing an empty file.

3. Final Answer:

   The log file path is incorrect or lacks write permission -> Option C

4. Quick Check:

   File permission issues cause empty logs [OK]

Hint: Check file path and permissions first [OK]

Common Mistakes:

• Blaming log level for missing error logs
• Ignoring file system permissions
• Assuming format string affects file writing

5. You want to design a logging strategy for a web app that must:
- Capture all errors and warnings
- Save detailed info logs only during debugging
- Send critical errors to an external alert system

Which combination of strategies best meets these needs?

hard

A. Use log level WARN for normal runs, switch to INFO for debugging, and configure alerts for ERROR logs

B. Always use log level INFO and save all logs to a single file without alerts

C. Use log level ERROR only and ignore INFO and WARN logs

D. Send all logs to the alert system regardless of level

Solution

1. Step 1: Match log levels to requirements

   WARN level captures warnings and errors; INFO level includes detailed info logs for debugging.

2. Step 2: Configure alerting for critical errors

   Setting alerts on ERROR logs ensures critical issues trigger notifications externally.

3. Step 3: Evaluate options

   Use log level WARN for normal runs, switch to INFO for debugging, and configure alerts for ERROR logs matches all requirements: dynamic log level and alerting. Others miss one or more needs.

4. Final Answer:

   Use log level WARN for normal runs, switch to INFO for debugging, and configure alerts for ERROR logs -> Option A

5. Quick Check:

   Dynamic levels + alerts = Use log level WARN for normal runs, switch to INFO for debugging, and configure alerts for ERROR logs [OK]

Hint: Adjust log level by mode and alert on errors [OK]

Common Mistakes:

• Logging everything all the time wastes resources
• Ignoring alerting for critical errors
• Using too high log level and missing info logs