Bird
Raised Fist0

Design a Real-Time Analytics Pipeline - Kafka, Flink, OLAP, Dashboard

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Design a Real-Time Analytics Pipeline - Kafka, Flink, OLAP, Dashboard
Design a scalable, low-latency real-time analytics pipeline using Kafka, Flink, OLAP stores, and dashboards
hardModern Infrastructure DesignFreq: high
Prerequisites:Message QueuesPub-Sub PatternStream ProcessingOLAP Databases
⚙️
Functional Requirements
  • Ingest event streams from multiple producers with at least 2 billion daily active users (DAU)
  • Process and aggregate events in real-time with end-to-end latency under 5 seconds
  • Store aggregated metrics in an OLAP store optimized for fast ad-hoc queries and dashboard refresh
  • Provide a dashboard API that supports sub-100ms P99 query latency for common analytics queries
  • Support flexible time window aggregations (e.g., 1 min, 5 min, hourly) and dimension filtering
  • Handle late-arriving and out-of-order events gracefully with event-time semantics
📊
Non-Functional Requirements
  • System must support 2 billion DAU generating 40 events per user per day
  • End-to-end processing latency (event ingestion to dashboard update) under 5 seconds
  • Dashboard query P99 latency under 100ms for common queries
  • System availability SLA of 99.99% with automated failover
  • Data retention of raw events for 7 days and aggregated data for 1 year
❓
Clarifying Questions to Ask
  • What is the expected event size and schema complexity?
  • Are events strictly append-only or can updates/deletes occur?
  • What is the expected query pattern on the dashboard? (e.g., top-k, drill-downs)
  • Is the system expected to support multi-tenant isolation or per-customer data?
  • What is the acceptable data freshness SLA for the dashboard?
  • Are there any constraints on cost or preferred cloud/on-premise deployment?
Out of Scope
✗ Designing the event producers or client SDKs
✗ User authentication and authorization for dashboard access
✗ Long-term cold storage or archival beyond 1 year
✗ Complex machine learning model training on event data
📌
Assumptions
  • 2 billion DAU generating 40 events per user per day
  • Average event size is 100 bytes
  • Retention of raw events for 7 days
  • Replication factor of 3 for durability
⚡
QPS Calculation
write qps2,000,000,000 × 40 / 86400 ≈ 925,926 msg/sec
peak write qpsAssuming 2× peak factor: 925,926 × 2 ≈ 1,851,852 msg/sec
read qpsDashboard queries estimated at 50,000 QPS peak
💾
Storage Estimate
daily raw storage2B × 40 × 100 bytes = 8,000,000,000,000 bytes = 8 TB/day
7 day raw retention8 TB × 7 = 56 TB
with replication56 TB × 3 = 168 TB total raw event storage
🖥️
Server Estimate
kafka brokersAssuming 50 MB/sec per broker throughput, 92.6 MB/sec / 50 ≈ 2 brokers minimum, add 3 for replication and headroom → 5 brokers
flink taskmanagersAssuming each TaskManager handles 50K msg/sec, 1,851,852 / 50,000 ≈ 37 TaskManagers needed
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          | Event Producers|
          +-------+--------+
                  |
                  v
          +-------+--------+
          |   Kafka Cluster |
          +-------+--------+
                  |
                  v
          +-------+--------+
          | Apache Flink   |
          | (Stream Proc)  |
          +-------+--------+
                  |
                  v
          +-------+--------+
          | OLAP Store     |
          | (ClickHouse)   |
          +-------+--------+
                  |
                  v
          +-------+--------+
          | Dashboard API  |
          +-------+--------+
                  |
                  v
          +-------+--------+
          | User Dashboards|
          +----------------+
🧩
Components
Kafka Cluster
Durable, scalable event ingestion and buffering layer
Why thisKafka supports 1M+ writes/sec per cluster with partitioning and replication, ensuring durability and ordering guarantees needed for 2B DAU scale
Why notRabbitMQ lacks horizontal scalability and throughput at this scale; Kinesis has higher cost and vendor lock-in
Apache Flink
Real-time stream processing with event-time semantics and exactly-once guarantees
Why thisFlink provides low-latency, stateful stream processing with built-in windowing and fault tolerance, critical for sub-5s end-to-end latency
Why notSpark Streaming has higher micro-batch latency unsuitable for real-time; Storm lacks native event-time support and state management
ClickHouse OLAP Store
High-performance columnar store optimized for fast analytical queries on aggregated data
Why thisClickHouse supports petabyte scale data, sub-100ms query latency on large datasets, and efficient compression for cost-effective storage
Why notPresto is a query engine, not a storage; Cassandra is optimized for OLTP, not OLAP queries
Dashboard API Service
Serves aggregated metrics to frontend dashboards with caching and query optimization
Why thisCustom API layer enables query caching, authorization, and query pattern optimization to meet 100ms P99 latency SLA
Why notDirect DB access from frontend risks exposing schema and causes high load spikes
Event Producers
Clients or services generating raw event data
Why thisProducers push events asynchronously to Kafka ensuring decoupling and scalability
Why notDirect writes to OLAP or processing layers would create tight coupling and bottlenecks
🔄
Data Flow
1
Event Producers → Kafka Cluster
Producers publish events asynchronously to Kafka topics partitioned by event type
Kafka partitions enable parallelism and ordering guarantees per partition
2
Kafka Cluster → Apache Flink
Flink consumes events from Kafka with checkpointing for fault tolerance
Flink uses event-time processing and watermarks to handle out-of-order events
3
Apache Flink → ClickHouse OLAP Store
Flink performs windowed aggregations and writes results to ClickHouse in near real-time
Aggregations reduce data volume and enable fast analytical queries
4
Dashboard API Service → ClickHouse OLAP Store
API queries aggregated metrics with filters and time ranges
API applies caching and query optimization to meet sub-100ms latency SLA
5
Dashboard API Service → User Dashboards
API serves aggregated analytics data to user dashboards
Dashboards refresh with near real-time data and support interactive queries

Practice

(1/5)
1. A browser needs to resolve a domain name to an IP address but wants to avoid repeated queries to external servers for the same domain within a short time. Which component in the DNS resolution chain primarily handles this optimization?
easy
A. The authoritative DNS server for the domain, which caches recent queries to speed up responses
B. The local DNS resolver (stub resolver) on the user's machine, which caches recent lookups to reduce network calls
C. The root DNS server, which maintains a cache of all domain-to-IP mappings globally
D. The recursive DNS server, which never caches results and always queries authoritative servers

Solution

  1. Step 1: Identify the role of the local DNS resolver

    The local DNS resolver (stub resolver) on the user's machine caches recent DNS lookups to avoid repeated network queries for the same domain within a short time.

  2. Step 2: Eliminate other options

    Authoritative DNS servers provide domain data but do not cache queries for clients. Root servers do not cache all mappings globally and are rarely queried directly. Recursive DNS servers do cache results to improve performance, so The recursive DNS server, which never caches results and always queries authoritative servers is incorrect.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Local resolver caches to reduce network calls [check]
Hint: Local resolver caches recent lookups to reduce queries [check]
Common Mistakes:
2. For delivering videos globally with low latency, should the system use a centralized origin server with global CDN caching, or a peer-to-peer (P2P) delivery network? Assume millions of users worldwide and variable network conditions.
medium
A. Use a centralized origin server with a global CDN to ensure consistent video availability and latency
B. Use P2P delivery combined with CDN fallback to optimize bandwidth and reliability
C. Use P2P delivery exclusively to reduce CDN costs and improve scalability
D. Use multiple regional origin servers without CDN to reduce latency

Solution

  1. Step 1: Evaluate delivery approaches for global scale and latency

    P2P (A) alone is unreliable due to variable user upload bandwidth and NAT traversal issues. Multiple regional origins without CDN (D) add complexity and don't scale well globally. P2P with CDN fallback (C) adds complexity and inconsistent performance.

  2. Step 2: Recognize CDN benefits

    A centralized origin with global CDN (B) provides consistent availability, low latency via edge caching, and simplifies management, making it the best fit for millions of users worldwide.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Global CDN ensures consistent low-latency delivery [check]
Hint: Global CDN with origin beats P2P for consistent delivery [check]
Common Mistakes:
3. Without the graph database's relationship index, what exactly breaks during a shortest path query between two nodes?
medium
A. The query silently returns empty results without any error indication
B. The query falls back to scanning all nodes and edges, causing severe latency spikes
C. The query returns incorrect paths due to stale data in the index
D. The query crashes the database server due to unhandled exceptions

Solution

  1. Step 1: Role of relationship index

    The index enables efficient lookup of edges; without it, traversal cannot proceed correctly.

  2. Step 2: Behavior without index

    Queries relying on the index return empty results silently because no edges are found, not errors or scans.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Missing index causes silent empty results, not crashes or scans [check]
Hint: Missing index -> silent empty results, no errors or scans [check]
Common Mistakes:
4. For a mobile chat app requiring exactly-once message delivery with minimal latency, which architecture is best suited?
medium
A. Use Kafka with transactional producers and consumers for exactly-once semantics
B. Use at-most-once delivery with client retries on failure
C. Use polling-based delivery with sequence numbers for ordering
D. Use a message queue with idempotent consumers and client-side deduplication

Solution

  1. Step 1: Evaluate Kafka transactions for mobile clients

    Kafka's transactional exactly-once requires stable connections and complex coordination, unsuitable for mobile clients with intermittent connectivity.

  2. Step 2: Assess idempotent consumers with client deduplication

    Idempotent consumers combined with client-side deduplication handle retries and duplicates gracefully, fitting mobile constraints and ensuring exactly-once delivery.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Idempotency + client deduplication -> practical exactly-once for mobile [check]
Hint: Mobile needs idempotency + deduplication, not Kafka transactions [OK]
Common Mistakes:
5. You must implement a distributed lock for a system with high contention and strict latency requirements. Which approach is best and why?
medium
A. Use Redis SETNX with a fixed TTL to minimize latency
B. Use a consensus-based lock service like etcd to ensure correctness despite latency
C. Use a database row lock with retries to handle contention
D. Use a local in-memory lock combined with periodic heartbeat checks

Solution

  1. Step 1: Evaluate Redis SETNX under high contention

    Redis SETNX with TTL (A) is fast but can cause split-brain and stale locks under network partitions, violating correctness.

  2. Step 2: Consider consensus-based lock service

    Consensus-based services like etcd (B) provide strong consistency and fault tolerance, ensuring correctness even if latency is higher.

  3. Step 3: Assess database row locks and local locks

    Database locks (C) have high latency and risk deadlocks; local locks (D) do not coordinate across nodes, failing correctness.

  4. Final Answer:

    Option B -> Option B

  5. Quick Check:

    Consensus ensures correctness under contention despite latency [check]
Hint: Consensus beats speed for correctness under contention [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard