Bird
Raised Fist0

Distributed Logging Architecture - Log Aggregation Pipeline Design

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Distributed Logging Architecture - Log Aggregation Pipeline Design
Design a scalable, durable, and low-latency log aggregation pipeline for billions of daily users
mediumMonitoring and ObservabilityFreq: high
Prerequisites:System Design FundamentalsCAP Theorem
⚙️
Functional Requirements
  • Ingest logs from 2 billion daily active users (DAU) across multiple platforms (mobile, web, backend).
  • Support real-time streaming of logs with sub-second latency for monitoring dashboards.
  • Persist logs durably for at least 30 days with configurable retention policies.
  • Provide query APIs for aggregated and raw log data with 100ms P99 latency for recent data.
  • Support multi-tenant isolation and access control for different product teams.
  • Allow schema evolution and support structured and unstructured log formats.
📊
Non-Functional Requirements
  • Handle peak write throughput of approximately 925K messages per second (derived from 2B DAU × 40 msgs/user/day).
  • Ensure 99.99% availability for log ingestion and query APIs.
  • Guarantee data durability with replication factor of 3 across data centers.
  • Support horizontal scalability to 10x growth without major redesign.
  • Maintain end-to-end latency under 1 second for log ingestion to availability in query.
❓
Clarifying Questions to Ask
  • What is the average and peak log message size in bytes?
  • Are logs structured (JSON) or unstructured (plain text)?
  • What is the expected query pattern: recent logs vs historical logs?
  • Is ordering of logs per user or service required?
  • What is the SLA for log availability after ingestion?
  • Are there any compliance or encryption requirements for logs?
Out of Scope
✗ Log analysis algorithms and anomaly detection.
✗ Long-term archival beyond 30 days (cold storage).
✗ User interface design for log visualization.
✗ Alerting and notification systems based on logs.
📌
Assumptions
  • 2 billion DAU generating 40 log messages per user per day.
  • Average log message size is 100 bytes.
  • Replication factor of 3 for durability.
  • Retention period of 30 days.
⚡
QPS Calculation
write qps2,000,000,000 users × 40 msgs/user/day ÷ 86,400 sec/day ≈ 925,926 msg/sec
peak write qpsAssuming 2× peak factor: 925,926 × 2 ≈ 1,851,852 msg/sec
read qpsAssuming 10% of write QPS for queries: 925,926 × 0.1 ≈ 92,593 queries/sec
💾
Storage Estimate
daily storage925,926 msg/sec × 100 bytes × 86,400 sec/day ≈ 8 TB/day uncompressed
retention storage8 TB/day × 30 days = 240 TB
with replication240 TB × 3 (replication) = 720 TB total storage
🖥️
Server Estimate
kafka brokersAssuming 100 MB/sec per broker throughput, need 95 MB/sec inbound → 2 brokers minimum, plan 10 for replication, partitioning, and peak load
storage nodesAssuming each Cassandra node stores 5 TB with replication, need 720 TB / 5 TB = 144 nodes
🏗️
Architecture Diagram
Architecture Diagram
User Devices / Services
       |
       v
Log Shippers (Fluentd/Fluent Bit)
       |
       v
Kafka Cluster (Partitioned Topics)
       |
       v
Stream Processing (Apache Flink)
       |
       v
Storage Cluster (Apache Cassandra)
       |
       v
Query API Servers (gRPC/REST)
       |
       v
Monitoring Dashboards / Clients
🧩
Components
Log Shippers (Fluentd/Fluent Bit)
Collect logs from various sources and forward to Kafka with buffering and backpressure handling.
Why thisFluentd/Fluent Bit support multiple input plugins, buffering, and reliable delivery to Kafka with minimal latency.
Why notCustom agents lack community support and require more maintenance; syslog is less flexible and lacks structured data support.
Kafka Cluster
Durable, partitioned, and replicated log ingestion buffer supporting high write throughput and ordered delivery per partition.
Why thisKafka scales horizontally to millions of messages per second with built-in replication and retention policies.
Why notRabbitMQ does not scale well for multi-terabyte retention and high throughput; Kinesis is cloud-specific and less flexible.
Stream Processing (Apache Flink)
Real-time processing and enrichment of logs, filtering, aggregation, and routing to storage.
Why thisFlink provides exactly-once semantics, low latency, and stateful stream processing at scale.
Why notSpark Streaming has higher latency and weaker exactly-once guarantees; custom batch jobs increase latency.
Storage Cluster (Apache Cassandra)
Store logs durably with high write throughput, horizontal scalability, and configurable TTL for retention.
Why thisCassandra’s wide-column model supports time-series data with tunable consistency and linear write scalability.
Why notPostgreSQL requires manual sharding and does not scale linearly at 1M+ writes/sec; HDFS has high write latency.
Query API Servers
Serve low-latency queries on recent logs and aggregated data with caching and multi-tenant access control.
Why thisStateless API servers behind load balancers enable horizontal scaling and isolation per tenant.
Why notDirect client access to storage nodes risks overload and security issues; monolithic services reduce scalability.
🔄
Data Flow
1
User Devices / Services → Log Shippers
Collect and buffer logs locally, batch and forward to Kafka
Buffers handle network spikes and offline scenarios
2
Log Shippers → Kafka Cluster
Publish logs to partitioned Kafka topics with replication
Partitions keyed by user_id or service to maintain ordering
3
Kafka Cluster → Stream Processing (Flink)
Consume logs for enrichment, filtering, and aggregation
Exactly-once processing ensures no duplicates
4
Stream Processing → Storage Cluster (Cassandra)
Write processed logs with TTL for retention
Writes are partitioned by (user_id, date) for efficient queries
5
Query API Servers → Storage Cluster
Serve queries for raw and aggregated logs
Cache recent queries to reduce load
6
Query API Servers → Monitoring Dashboards / Clients
Return query results with low latency
Supports multi-tenant access control

Practice

(1/5)
1. A global e-commerce platform wants to route users to the nearest data center to reduce latency. Which component primarily handles this geographic traffic distribution?
easy
A. GeoDNS service that resolves domain names to IPs based on user location
B. A centralized load balancer that routes traffic based on server health checks
C. A CDN that caches static content globally without DNS involvement
D. An application-level proxy that inspects HTTP headers to route requests

Solution

  1. Step 1: Identify GeoDNS role

    GeoDNS resolves DNS queries differently based on the client's geographic location, enabling routing to the nearest data center.

  2. Step 2: Eliminate other options

    A centralized load balancer (A) is a single point and doesn't handle geo-based DNS resolution. CDN (C) caches content but doesn't route DNS queries. Application-level proxy (D) routes after DNS resolution, not at DNS level.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    GeoDNS routes based on client location via DNS resolution [check]
Hint: GeoDNS routes users by location at DNS level [check]
Common Mistakes:
2. In a load balancer using Weighted Round Robin, at step 5 the balancer receives a new request. The current server weights are: Server A=5, Server B=3, Server C=2. Which server does the balancer select next?
easy
A. Server A, because Weighted Round Robin cycles through servers proportionally to their weights
B. Server B, because it has fewer active connections than Server A
C. Server C, because it was least recently selected
D. Server A, because it has the highest static weight

Solution

  1. Step 1: Understand Weighted Round Robin operation

    Weighted Round Robin cycles through servers, selecting each server multiple times proportional to its weight before moving to the next.

  2. Step 2: Apply weights to selection

    Server A with weight 5 will be selected more frequently than B or C. At step 5, the balancer continues cycling respecting weights.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Weighted Round Robin respects weights in cycling, not just highest weight always [check]
Hint: Weighted Round Robin cycles servers proportional to weights, not just highest weight [check]
Common Mistakes:
3. If the Metadata Server crashes before acknowledging a file rename operation, what is the most likely failure observed by the client?
medium
A. Client's write operations to the file silently fail without notification
B. Client receives an error and the file remains under the old name with no partial rename
C. Client sees the file renamed but data blocks lost
D. Client sees the file under both old and new names due to partial metadata update

Solution

  1. Step 1: Understand rename operation atomicity

    Rename involves updating metadata entries; if crash occurs mid-update, partial state may persist.

  2. Step 2: Trace failure effect

    Partial metadata update can cause both old and new filenames to appear, confusing clients.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Partial metadata updates cause duplicate filenames visible to client [check]
Hint: Partial metadata update causes duplicate filenames visible [check]
Common Mistakes:
4. If the health check component is removed from a load balancer using Least Connections, what failure occurs?
medium
A. The load balancer continues to distribute requests evenly, but some servers may be down causing request failures
B. The load balancer switches to Round Robin automatically, losing connection count accuracy
C. The load balancer caches server states incorrectly, causing stale routing decisions
D. The load balancer sends requests to unhealthy servers, causing increased latency and failed connections without fallback

Solution

  1. Step 1: Role of health checks in Least Connections

    Health checks detect server availability; without them, the balancer cannot exclude unhealthy servers.

  2. Step 2: Behavior without health checks

    Least Connections relies on accurate connection counts and server health; without health checks, it cannot trust connection data and typically falls back to Round Robin.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Without health checks, Least Connections falls back to Round Robin losing load awareness [check]
Hint: No health checks -> Least Connections falls back to Round Robin [check]
Common Mistakes:
5. If the prefix hash map component fails and becomes unavailable, what is the immediate impact on the autocomplete system's behavior?
medium
A. The system returns no results for prefix queries until the prefix hash map recovers
B. The Trie component automatically handles prefix queries without any degradation in performance
C. Autocomplete queries fall back to Elasticsearch but with increased latency and less precise prefix matching
D. User input is cached locally on the client until the prefix hash map is restored

Solution

  1. Step 1: Understand component roles

    The prefix hash map is an optimization for O(1) prefix lookups, but the Trie is the primary data structure for prefix matching.

  2. Step 2: Trace failure impact

    If the prefix hash map fails, the system falls back to the Trie for prefix queries, which still works but may have slightly higher latency.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Trie fallback ensures prefix queries continue despite prefix hash map failure [check]
Hint: Trie fallback keeps prefix search alive if prefix hash fails [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard