Bird
Raised Fist0

Latency vs Throughput - The Core Trade-off Every Design Must Balance

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Latency vs Throughput - The Core Trade-off Every Design Must Balance
45 min · senior
senior

You are designing a messaging backend for a startup with 50 million daily active users (DAU). The system must deliver messages with end-to-end encryption, support offline message delivery, and maintain a P99 latency under 200ms for online users. The business requires that message delivery is reliable and scalable to peak loads, which can reach 1 million messages per second. The system must also handle user reconnections gracefully without overwhelming backend resources.

Constraints
  • End-to-end encryption must be enforced for all messages at rest and in transit
  • Offline message delivery with guaranteed eventual consistency
  • P99 end-to-end latency under 200ms for online message delivery
  • Peak throughput of 1 million messages per second
  • Support for efficient client reconnection and message sync without excessive load
📋
Guided Steps
1/5
Step 1
Choosing the Messaging Backbone
What messaging infrastructure will you use to handle 1 million messages per second with ordered delivery per conversation?
💡 Hint
Consider partitioning strategies that preserve message order within conversations and scale linearly with throughput.
Key Decision
Use Apache Kafka with conversation_id as the partition key - ensures all messages in a conversation land on the same partition for ordered processing and linear scalability.
Interviewer Expectation
Mid-level: Names Kafka for high throughput. Senior: Explains why conversation_id is chosen as partition key over sender_id to maintain order and avoid cross-partition coordination.
⚠️Common Wrong Answer
Using RabbitMQ or Redis Pub/Sub without partitioning - these systems do not scale linearly at 1M msg/sec and cannot guarantee ordered delivery per conversation at this scale.
Step 2
Data Storage for Encrypted Messages
How will you store encrypted messages to support offline delivery and efficient sync with low latency?
💡 Hint
Look for a storage system optimized for high write throughput and range queries by conversation for cursor-based sync.
Key Decision
Use Apache Cassandra with partition_key=conversation_id - LSM-tree writes scale linearly, and range queries enable efficient cursor-based message sync on reconnect.
Interviewer Expectation
Mid-level: Suggests a NoSQL store. Senior: Justifies Cassandra’s partition key choice and rejects relational DBs due to write amplification and scaling bottlenecks.
⚠️Common Wrong Answer
Using a relational database with conversation_id indexed - leads to write contention and slow range scans at 1M msg/sec scale.
Step 3
Balancing Latency and Throughput in Delivery
How do you design the delivery path to meet P99 latency under 200ms while handling peak throughput?
💡 Hint
Consider separating online fast-path delivery from offline batch delivery and using caching to reduce backend load.
Key Decision
Implement a delivery worker tier that reads from Kafka partitions and pushes messages via WebSocket for online users, falling back to offline storage sync for disconnected users; use Redis as a presence cache to quickly determine user online status.
Interviewer Expectation
Mid-level: Describes WebSocket delivery. Senior: Explains Redis TTL-based presence detection and how it prevents unnecessary backend load from offline users.
⚠️Common Wrong Answer
Polling clients frequently for messages - causes backend overload and violates latency SLA at 50M DAU.
Step 4
Handling Client Reconnection and Sync
How will clients efficiently sync missed messages after reconnecting without overwhelming the system?
💡 Hint
Use cursor-based incremental sync and avoid full data scans or repeated polling.
Key Decision
Clients send last received message offset; backend uses Cassandra range queries by conversation_id to fetch only missed messages, avoiding full scans and reducing load.
Interviewer Expectation
Mid-level: Mentions offset-based sync. Senior: Details how Cassandra’s partitioning enables O(1) range scans and why full table scans or timestamp queries are inefficient.
⚠️Common Wrong Answer
Full message list fetch on reconnect - causes high latency and backend overload at scale.
Step 5
Ensuring End-to-End Encryption and Security
How do you enforce end-to-end encryption while still enabling offline delivery and message sync?
💡 Hint
Consider client-side encryption and how metadata is handled separately from encrypted payloads.
Key Decision
Clients encrypt messages locally; backend stores only ciphertext and metadata (conversation_id, offsets) in Cassandra; delivery workers do not decrypt messages, preserving end-to-end encryption.
Interviewer Expectation
Mid-level: Mentions encryption at rest and in transit. Senior: Explains client-side encryption and separation of metadata for routing without compromising security.
⚠️Common Wrong Answer
Encrypting messages only on transport layer - backend can read plaintext, violating end-to-end encryption requirement.
🔌
API Design Challenge
Sketch 3 core API endpoints for the messaging system that balance latency and throughput under the given constraints.
Hints
  • How does the client send a message securely and efficiently?
  • How does the client synchronize missed messages after reconnecting?
  • How does the recipient acknowledge message receipt to ensure reliability?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of Kafka with conversation_id partition key for ordered, scalable ingestion
  • Use of Cassandra with conversation_id partition key for efficient storage and sync
  • Presence detection with Redis TTL cache to reduce backend load
  • WebSocket-based online delivery path meeting P99 latency < 200ms
  • Cursor-based incremental sync to avoid full scans on reconnect
  • Client-side end-to-end encryption preserving message confidentiality
Good to Have - 30%
  • Separation of online and offline delivery paths to balance latency and throughput
  • API design covering send, sync, and acknowledge with security considerations
  • Discussion of failure modes such as Kafka partition leader failure or Cassandra node outages
Bonus - 10%
  • Use of Kafka consumer groups for load balancing delivery workers
  • Optimizations like batching messages per conversation to reduce overhead
  • Metrics and monitoring strategies for latency and throughput SLAs
Anti-patterns to Avoid
HTTP polling at 50M DAU: 50 million reconnects per minute just to check for messages - collapses under load and violates latency SLA
Using RabbitMQ or Redis Pub/Sub without partitioning: does not scale linearly at 1M msg/sec and cannot guarantee ordered delivery per conversation
Encrypting messages only on transport layer: backend can read plaintext, violating end-to-end encryption requirement
Full message list fetch on reconnect: causes high latency and backend overload at scale
Using relational databases with conversation_id indexed: leads to write contention and slow range scans at 1M msg/sec scale
✅
Model Answer
▼ Reveal

Design a messaging backend that balances latency and throughput by using Apache Kafka for ordered, scalable ingestion, Apache Cassandra for efficient encrypted message storage and sync, Redis for presence detection, and WebSocket delivery for low-latency online message push. Client-side encryption ensures end-to-end security while offline delivery and incremental sync handle disconnected users efficiently.

Key Decisions
  • Use Apache Kafka with conversation_id as partition key to guarantee ordered message ingestion and linear scalability at 1M msg/sec
  • Store encrypted messages in Apache Cassandra partitioned by conversation_id to support high write throughput and efficient cursor-based range queries for sync
  • Implement Redis TTL-based presence cache to quickly determine user online status and reduce backend load from offline users
  • Separate online fast-path delivery via WebSocket from offline batch sync to meet P99 latency under 200ms while handling peak throughput
  • Clients send last received message offset to backend, which uses Cassandra range queries to fetch only missed messages, avoiding full scans
  • Enforce client-side end-to-end encryption so backend stores only ciphertext and metadata, preserving message confidentiality
  • Apache Kafka with conversation_id as partition key for ordered, scalable message ingestion
  • Apache Cassandra with conversation_id partition key for high write throughput and efficient range queries
  • Redis with TTL-based presence cache to quickly determine online/offline status and reduce backend load
  • WebSocket delivery workers consuming Kafka partitions for low-latency online message push
  • Cursor-based incremental sync using Cassandra range scans to efficiently fetch missed messages on reconnect
  • Client-side end-to-end encryption ensuring backend stores only ciphertext and metadata, preserving privacy
  • Separation of online fast-path delivery and offline batch sync to balance latency and throughput
  • API design supporting sendMessage, syncMessages, and acknowledgeMessage endpoints for secure, reliable communication

Practice

(1/5)
1. In the request flow, at Step 4 the Context Manager retrieves the conversation history before invoking the LLM. What happens if the context window exceeds the maximum token limit at this step?
easy
A. The Context Manager truncates the oldest messages to fit the token limit before sending to the LLM
B. The Context Manager sends the entire conversation history, causing the LLM to reject the request
C. The Context Manager splits the conversation into multiple requests to the LLM in parallel
D. The Context Manager caches the full history and waits for manual intervention

Solution

  1. Step 1: Understand Context Manager role at Step 4

    It must prepare input within the LLM's token window limit to avoid rejection.

  2. Step 2: Analyze behavior on exceeding token limit

    Sending full history (B) causes LLM errors; splitting requests (C) breaks context continuity; caching and manual intervention (D) is impractical. Truncation (A) preserves recent context within limits.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Context Manager truncates oldest messages to fit token window [check]
Hint: Truncate oldest context to fit token window, not entire history [check]
Common Mistakes:
2. A typeahead system must handle millions of concurrent prefix queries with low latency and high throughput. Which component is best suited to serve these prefix lookups efficiently at scale?
easy
A. A distributed in-memory Trie cache sharded by prefix hash
B. A message queue to buffer prefix queries before processing
C. A full-text search engine like Elasticsearch without caching
D. A relational database with indexed LIKE queries on the prefix column

Solution

  1. Step 1: Identify prefix lookup requirements

    Prefix queries require fast, low-latency lookups optimized for string prefixes, which relational DB LIKE queries are inefficient at scale.

  2. Step 2: Evaluate component suitability

    A distributed in-memory Trie cache supports efficient prefix matching and can be sharded by prefix hash for scalability, unlike message queues or full-text search without caching.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Distributed Trie cache -> efficient prefix lookups at scale [check]
Hint: Trie caches excel at prefix queries, unlike DB LIKE or queues [check]
Common Mistakes:
3. A web application requires users to authenticate with multi-factor authentication (MFA) before accessing sensitive data. Which component is primarily responsible for enforcing MFA in this scenario?
easy
A. The Database, because it stores user credentials and MFA tokens
B. The API Gateway, because it routes requests and can reject unauthorized traffic
C. The Identity Provider (IdP), because it manages user credentials and enforces MFA policies
D. The Load Balancer, because it distributes authentication requests across servers

Solution

  1. Step 1: Identify the role of the Identity Provider

    The IdP manages user credentials and enforces authentication policies including MFA.

  2. Step 2: Eliminate other components

    The API Gateway routes traffic but does not enforce MFA; the Database stores data but does not enforce policies; the Load Balancer distributes requests but does not handle authentication.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    IdP enforces MFA policies, not routing or storage components [check]
Hint: MFA enforcement belongs to the Identity Provider [check]
Common Mistakes:
4. If the cache warming service crashes after partially warming keys but before marking completion, what is the immediate failure observed during launch?
medium
A. The cache contains stale data from previous runs causing incorrect responses
B. The cache warming service automatically retries from the start, causing duplicate warming and wasted resources
C. The system falls back to a read-through cache mode with no performance impact
D. Clients experience cache misses for keys not warmed, causing increased latency and possible DB overload

Solution

  1. Step 1: Trace failure impact of partial warming crash

    Without completion mark, warming restarts causing duplicate warming and resource waste.

  2. Step 2: Eliminate other options

    Stale data unlikely if warming overwrites; cache misses occur but duplicate warming is immediate failure; fallback read-through causes latency, not immediate failure.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Crash without completion causes duplicate warming retries [check]
Hint: Crash without completion mark causes duplicate warming retries [check]
Common Mistakes:
5. Without the retention policy enforcement component in a time-series database, what exact failure occurs?
medium
A. Queries return stale data because old data is never deleted
B. Storage fills up causing ingestion to block and new data to be lost
C. Clients receive inconsistent query results due to missing indexes
D. Data is corrupted due to overlapping retention windows

Solution

  1. Step 1: Understand retention policy role

    Retention policies delete or downsample old data to manage storage and maintain data consistency.

  2. Step 2: Analyze failure without retention enforcement

    Without enforcement, overlapping retention windows cause data duplication or corruption during compaction/downsampling. Storage filling (B) is secondary and delayed. Stale queries (A) and missing indexes (D) are unrelated.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Retention enforcement prevents data corruption from overlapping windows [check]
Hint: Retention policies prevent data overlap and corruption [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard