Bird
Raised Fist0

Distributed Logging Architecture - Log Aggregation Pipeline Design

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/5
Log Shippers (Fluentd/Fluent Bit)

Collect log event from application and forward to Kafka

Log shipper batches and sends logs to Kafka topic

Protocol:Kafka Producer API
BeforeLog event exists only on application host
AfterLog event is sent and durably stored in Kafka topic
Kafka Cluster

Store log event in partitioned topic

Kafka appends log event to partition log with replication

Protocol:Kafka Internal Storage
BeforeLog event received from shipper but not yet stored
AfterLog event durably stored and available for consumers
Stream Processing (Apache Flink)

Consume log event from Kafka and enrich/process it

Flink applies transformations and filters on log stream

Protocol:Kafka Consumer API
BeforeLog event is stored in Kafka topic
AfterProcessed log event ready to be stored in Cassandra
Storage Cluster (Apache Cassandra)

Write processed log event to distributed storage

Cassandra stores log event partitioned by timestamp and source

Protocol:CQL (Cassandra Query Language)
BeforeProcessed log event exists in Flink output stream
AfterLog event persisted in Cassandra cluster for querying
Query API Servers

Receive user query and fetch logs from Cassandra

API server executes CQL queries to retrieve logs

Protocol:CQL over TCP
BeforeUser query received, no logs fetched yet
AfterQuery results returned to user with requested logs
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────────────────────┐
│ Log Shippers                  │
│ (Fluentd/Fluent Bit)          │
└───────────────┬───────────────┘
                │ Kafka Producer
                ▼
┌───────────────────────────────┐
│ Kafka Cluster                 │
│ (Distributed Log Queue)       │
└───────────────┬───────────────┘
                │ Kafka Consumer
                ▼
┌───────────────────────────────┐
│ Stream Processing             │
│ (Apache Flink)                │
└───────────────┬───────────────┘
                │ Write Processed Logs
                ▼
┌───────────────────────────────┐
│ Storage Cluster               │
│ (Apache Cassandra)            │
└───────────────┬───────────────┘
                │ Query Requests
                ▼
┌───────────────────────────────┐
│ Query API Servers             │
│ (Serve Log Queries)           │
└───────────────────────────────┘
Design Rationale

This distributed logging architecture ingests logs via Log Shippers that forward events to a Kafka Cluster, which acts as a durable, partitioned log queue. Apache Flink consumes logs from Kafka, performing real-time stream processing such as filtering and enrichment. The processed logs are then stored in a scalable Cassandra cluster optimized for time-series data. Query API Servers serve user requests by querying Cassandra, enabling efficient retrieval of logs. This design decouples ingestion, processing, and querying, ensuring high throughput, fault tolerance, and scalability across components.

Primary Focus

✓

Practice

(1/5)
1. Without the retry queue component in the payment gateway's webhook delivery system, what exact failure occurs when a merchant's endpoint is temporarily down?
medium
A. Webhook notifications are retried immediately in a tight loop, overwhelming the merchant server
B. Webhook delivery blocks the main payment processing thread until the merchant recovers
C. Webhook notifications are queued in memory indefinitely, causing memory leaks and crashes
D. Webhook notifications are lost permanently with no retry, causing inconsistent merchant state

Solution

  1. Step 1: Understand retry queue role

    The retry queue buffers failed webhook notifications for asynchronous retries, ensuring eventual delivery.

  2. Step 2: Consequence of missing retry queue

    Without it, failed webhooks are dropped or lost, causing permanent notification loss and inconsistent merchant state.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Missing retry queue -> permanent webhook loss on failure [check]
Hint: Retry queue prevents permanent webhook loss [check]
Common Mistakes:
2. Without Kafka in a messaging system, what exactly breaks when Alice sends a message to Bob and Bob's server crashes before acknowledging receipt?
medium
A. Alice's send blocks until Bob responds; if Bob's server crashes, message is lost with no retry
B. Alice's message is queued in memory and delivered later without loss
C. Alice's client retries sending indefinitely until Bob's server recovers
D. Messages are encrypted incorrectly causing data corruption

Solution

  1. Step 1: Understand Kafka's role in decoupling sender and receiver

    Kafka buffers messages durably, allowing sender to proceed without waiting for receiver.

  2. Step 2: Trace failure without Kafka

    Without Kafka, sender must synchronously wait for receiver acknowledgment; if Bob's server crashes, no durable store exists, causing message loss and blocking sender.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Without Kafka, synchronous send blocks and loses message if receiver crashes [check]
Hint: No Kafka means blocking send and lost messages on receiver crash [check]
Common Mistakes:
3. Which monitoring strategy best prevents failover incidents like GitHub's database crash by detecting anomalies early?
medium
A. Periodic manual health checks by DBAs
B. Weekly backup verification reports
C. Real-time metrics with alerting on leader election delays
D. Client-side error logging without centralized monitoring

Solution

  1. Step 1: Importance of monitoring leader election

    Real-time metrics detect leader election delays indicating failover issues early.

  2. Step 2: Evaluate other options

    Manual checks (A) are slow; backup reports (C) don't catch live failover issues; client logs (D) lack centralized alerting.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Real-time alerting enables proactive failover management [check]
Hint: Real-time alerts catch failover delays early
Common Mistakes:
4. At 10x scale, which component in a message processing system with Dead Letter Queues is most likely to become the bottleneck first, and why?
hard
A. The monitoring system, because alert volume grows exponentially with message failures
B. The main message broker, because partition count limits throughput and causes uneven load
C. The retry worker pool, because retrying messages requires synchronous processing and cannot be parallelized
D. The Dead Letter Queue storage, because it accumulates failed messages faster than they can be retried or inspected

Solution

  1. Step 1: Analyze scaling impact on DLQ storage

    At 10x scale, failed messages accumulate rapidly in DLQ storage, which may not scale as fast as main queue throughput.

  2. Step 2: Evaluate other components

    Main broker (A) can scale with partitions; retry workers (C) can be scaled horizontally; monitoring (D) is less critical bottleneck.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    DLQ storage fills faster than retries can process at 10x scale [check]
Hint: DLQ storage fills first at scale, not broker or workers [OK]
Common Mistakes:
5. At 10x scale, which component in a push-based metrics collection system is most likely to fail first, and why?
hard
A. The aggregation workers, because CPU-intensive aggregation cannot scale horizontally
B. The storage backend, because it cannot handle the increased write throughput
C. The Kafka cluster, because partition count limits throughput and causes broker overload
D. The metric agents, because they overwhelm network bandwidth with frequent pushes

Solution

  1. Step 1: Analyze storage backend scaling

    Storage can be scaled horizontally with sharding; unlikely first failure.

  2. Step 2: Analyze Kafka cluster scaling

    Kafka scales with partitions but requires careful tuning; not first failure at 10x.

  3. Step 3: Analyze aggregation workers

    Aggregation workers scale horizontally; CPU load manageable with more instances.

  4. Step 4: Analyze metric agents

    Agents push metrics frequently, increasing network usage exponentially; network bandwidth at agents is limited and saturates first.

  5. Final Answer:

    Option D -> Option D

  6. Quick Check:

    Agents saturate network bandwidth first at 10x scale [check]
Hint: Agents saturate network bandwidth before backend bottlenecks [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard