Bird
Raised Fist0

Latency vs Throughput - The Core Trade-off Every Design Must Balance

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Latency vs Throughput - The Core Trade-off Every Design Must Balance
Understanding and balancing latency and throughput trade-offs in distributed system design
mediumCore Distributed Systems ConceptsFreq: high
Prerequisites:System Design FundamentalsCAP Theorem
⚙️
Functional Requirements
  • Support high throughput data ingestion with minimal latency
  • Provide low-latency read access for user queries
  • Ensure data durability and consistency under load
  • Allow configurable trade-offs between latency and throughput per use case
  • Support horizontal scaling to handle growth in users and data volume
📊
Non-Functional Requirements
  • Support 2 billion daily active users (DAU)
  • Handle peak write throughput of approximately 925,000 messages per second
  • Achieve 99.99% availability SLA
  • Maintain 100ms P99 read latency under peak load
  • Data retention of 30 days with replication factor of 3
❓
Clarifying Questions to Ask
  • What is the expected message size and variability?
  • Are reads or writes prioritized for latency or throughput?
  • Is eventual consistency acceptable or is strong consistency required?
  • What is the acceptable data loss threshold during failures?
  • Are there specific SLAs for peak vs off-peak hours?
  • What is the expected read-to-write ratio?
Out of Scope
✗ End-user application UI design
✗ Detailed security and authentication mechanisms
✗ Offline batch analytics processing
✗ Cross-region multi-master replication
📌
Assumptions
  • 2 billion DAU
  • 40 messages per user per day
  • 100 bytes average message size
  • Retention period of 30 days
  • Replication factor of 3
⚡
QPS Calculation
write qps2,000,000,000 × 40 / 86,400 ≈ 925,926 messages/sec
peak write qpsAssuming 2× peak factor → 1,851,852 messages/sec
read qpsAssuming 5 reads per write → 9,259,260 reads/sec
💾
Storage Estimate
daily storage gb(925,926 msg/sec × 100 bytes × 86,400 sec) / (1024³) ≈ 7.45 TB/day
retention storage gb7.45 TB/day × 30 days = 223.5 TB
with replication223.5 TB × 3 = 670.5 TB total storage
🖥️
Server Estimate
write pathKafka cluster with 100 brokers (each 20K msg/sec capacity) to handle 1.85M peak writes/sec
read pathCassandra cluster with 200 nodes (each 50K reads/sec) to handle 9.25M reads/sec
🏗️
Architecture Diagram
Architecture Diagram
          +-------------+
          |   Clients   |
          +------+------+
                 |
          +------+-------+
          |  API Gateway |
          +------+-------+
                 |
          +------+-------+
          | Kafka Cluster|
          +------+-------+
                 |
          +------+-------+
          | Cassandra DB|
          +------+-------+
                 |
          +------+-------+
          | Cache Layer |
          +------+-------+
                 |
          +------+-------+
          |Delivery Svc |
          +--------------+
🧩
Components
API Gateway
Handles client requests, performs authentication, throttling, and routes writes to Kafka and reads to cache or DB
Why thisNginx with Lua scripting supports high concurrency and flexible routing with low latency
Why notAWS API Gateway adds latency and cost at 925K QPS; Envoy is heavier and more complex
Kafka Cluster
Durable, distributed write log to ingest high throughput messages asynchronously
Why thisKafka scales linearly to millions of writes/sec with partitioning and provides strong durability guarantees
Why notRabbitMQ does not scale linearly at this throughput and has higher latency under load
Cassandra DB
Distributed wide-column store for scalable, low-latency reads and writes with tunable consistency
Why thisCassandra supports linear horizontal scaling and configurable consistency levels for latency vs throughput trade-offs
Why notPostgreSQL requires manual sharding and cannot handle 1M+ writes/sec without bottlenecks
Cache Layer
In-memory caching (Redis Cluster) to serve hot reads with sub-100ms latency
Why thisRedis Cluster supports sharding and TTL expiration natively, reducing DB load and improving read latency
Why notMemcached lacks persistence and advanced eviction policies needed for this scale
Delivery Service
Processes messages from Cassandra and cache to deliver to end-users or downstream systems
Why thisCustom microservice with backpressure and batching optimizes throughput and latency for delivery
Why notUsing direct DB triggers or polling causes unpredictable latency and throughput bottlenecks
🔄
Data Flow
1
Clients → API Gateway
Send write/read requests
API Gateway authenticates and routes requests
2
API Gateway → Kafka Cluster
Writes messages asynchronously
Decouples ingestion from storage for high throughput
3
Kafka Cluster → Cassandra DB
Consume and persist messages
Ensures durable storage with replication
4
API Gateway → Cache Layer
Serve read requests from Redis
Low latency reads for hot data
5
Cache Layer → Cassandra DB
Fallback for cache misses
Ensures data availability
6
Delivery Service → Clients/Downstream
Deliver messages
Optimized for throughput with batching

Practice

(1/5)
1. A distributed key-value store must evenly distribute keys across nodes and handle node failures without massive data reshuffling. Which component primarily enables this?
easy
A. Gossip protocol to propagate membership and failure information
B. Vector clocks to resolve conflicting versions of data
C. Consistent hashing to assign keys to nodes with minimal remapping
D. Quorum-based replication to ensure data durability

Solution

  1. Step 1: Identify the role of consistent hashing

    Consistent hashing distributes keys to nodes such that when nodes join or leave, only a small subset of keys need reassignment, minimizing reshuffling.

  2. Step 2: Eliminate other components

    Vector clocks handle version conflicts but do not distribute keys. Gossip protocol manages cluster membership but not key distribution. Quorum replication ensures durability but does not affect key assignment.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Consistent hashing minimizes key remapping on node changes [check]
Hint: Consistent hashing minimizes key remapping on node changes [OK]
Common Mistakes:
2. A live streaming platform must ingest RTMP streams from millions of broadcasters with minimal latency and reliably convert them to HLS segments for viewers. Which component is best suited to buffer and order incoming RTMP packets before transcoding?
easy
A. A relational database with ACID transactions to store RTMP packets
B. An in-memory cache like Redis with TTL eviction
C. A distributed message queue like Kafka partitioned by stream ID
D. A CDN edge server caching HLS segments

Solution

  1. Step 1: Identify buffering and ordering needs

    RTMP packets must be buffered and ordered per stream before transcoding to maintain low latency and correct playback order.

  2. Step 2: Evaluate component suitability

    A relational DB (A) is too slow for high-throughput packet buffering. Redis (C) lacks strong ordering guarantees and durability. CDN edge (D) caches output, not input packets. Kafka (B) supports partitioning by stream ID, ensuring ordered, durable buffering.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Kafka partitions ensure ordered, durable buffering per stream [check]
Hint: Kafka partitions per stream ensure ordered, durable buffering [check]
Common Mistakes:
3. In the failover incident trace, at Step 4 the Database Proxy receives a client query after the primary node crashed. What state does the proxy enter before forwarding the query?
easy
A. Marks the primary as unavailable and routes query to the new leader after election
B. Returns an error to the client without retrying
C. Queues the query indefinitely until the old primary recovers
D. Immediately forwards query to the old primary node, causing timeout

Solution

  1. Step 1: Trace Database Proxy behavior at Step 4

    Upon detecting primary failure, the proxy marks it unavailable and waits for leader election to complete before routing queries to the new leader.

  2. Step 2: Eliminate other options

    Forwarding immediately to old primary (A) causes timeouts; queuing indefinitely (C) stalls clients; returning errors without retry (B) reduces availability.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Proxy reroutes queries only after leader election [check]
Hint: Proxy reroutes only after leader election completes
Common Mistakes:
4. For candidate-job matching, which architecture better supports real-time personalized recommendations with low latency and frequent profile updates?
medium
A. Precomputed static candidate-job similarity scores stored in a relational database
B. Batch processing with nightly offline model training and periodic reindexing
C. Online feature store with streaming updates feeding a real-time scoring service
D. Client-side filtering of job listings based on cached candidate preferences

Solution

  1. Step 1: Analyze real-time personalization needs

    Low latency and frequent profile updates require online, streaming data processing.

  2. Step 2: Evaluate architectures

    Batch (A) is too stale; static precomputed scores (C) can't adapt quickly; client-side filtering (D) lacks server-side intelligence. Online feature store with streaming (B) supports fresh data and low latency scoring.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Streaming updates + real-time scoring -> fresh personalized matches [check]
Hint: Streaming + online features enable fresh real-time recommendations [check]
Common Mistakes:
5. For delivering real-time location updates to mobile clients, which approach is better considering intermittent mobile network connectivity and battery constraints?
medium
A. Use WebSocket connections with automatic reconnection and heartbeat pings
B. Use persistent gRPC streaming connections to push updates continuously
C. Use HTTP long polling to simulate real-time updates
D. Use periodic client polling every few seconds to fetch updates

Solution

  1. Step 1: Evaluate persistent connection options

    gRPC streaming is efficient but less supported on mobile platforms and can drain battery.

  2. Step 2: Compare WebSocket and polling

    WebSocket supports bidirectional communication with reconnection and heartbeats, balancing real-time and battery use better than polling.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    WebSocket balances real-time updates and mobile constraints [check]
Hint: WebSocket with reconnection beats gRPC streaming on mobile [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard