Bird
Raised Fist0

Kafka Architecture - Partitions, Consumer Groups, Offsets, Retention Policy

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Kafka Architecture - Partitions, Consumer Groups, Offsets, Retention Policy
45 min · senior
senior

You are designing the core messaging backbone for a startup with 50 million daily active users (DAU). The system must support real-time message delivery with P99 latency under 200ms, guarantee message ordering per conversation, and ensure data encryption at rest and in transit. Offline message delivery is required so users can receive messages sent while they were disconnected. The system must scale to handle peak throughput of 1 million messages per second. Retention policy must balance storage costs with user experience, allowing message replay within 7 days.

Constraints
  • End-to-end encryption required for all messages at rest and in transit
  • Support offline message delivery with exactly-once semantics
  • P99 latency for message delivery under 200ms at peak load
  • Retention policy configurable per topic with default 7 days
  • Scale to 1 million messages per second peak throughput
📋
Guided Steps
1/5
Step 1
Partitioning Strategy
How would you design the Kafka partitioning scheme to ensure message ordering and load balancing?
💡 Hint
Consider using a key that groups related messages to the same partition for ordering guarantees.
Key Decision
Use conversation_id as Kafka partition key - ensures all messages in a conversation land on the same partition, preserving order and enabling consumer affinity.
Interviewer Expectation
Mid-level: Names Kafka partitions and mentions ordering. Senior: Explains why conversation_id is chosen over sender_id or random keys to avoid out-of-order delivery and uneven load.
⚠️Common Wrong Answer
Using sender_id as partition key - causes messages from the same conversation to scatter across partitions, breaking ordering guarantees.
Step 2
Consumer Group and Offset Management
How will you design consumer groups and offset tracking to support exactly-once processing and offline delivery?
💡 Hint
Think about how Kafka consumer groups commit offsets and how to handle reprocessing after offline periods.
Key Decision
Use Kafka consumer groups with manual offset commits stored in Kafka’s __consumer_offsets topic, enabling consumers to resume from last committed offset for exactly-once processing and offline sync.
Interviewer Expectation
Mid-level: Describes consumer groups and offset commits. Senior: Details manual offset commit strategy, offset retention, and how it supports offline delivery and reprocessing without duplicates.
⚠️Common Wrong Answer
Auto-commit offsets immediately - risks message loss or duplicates during consumer crashes or offline periods.
Step 3
Retention Policy Design
How would you configure Kafka retention policies to balance storage costs and user experience for message replay?
💡 Hint
Consider retention time and size limits per topic and how they affect message availability for offline users.
Key Decision
Configure topic-level retention with time-based retention of 7 days and size limits per partition, ensuring messages are available for offline replay while controlling storage costs.
Interviewer Expectation
Mid-level: Mentions retention time configuration. Senior: Explains trade-offs between time vs size retention, impact on offline delivery, and how compaction is not suitable for ephemeral chat messages.
⚠️Common Wrong Answer
Using log compaction for chat topics - leads to message loss as only latest key versions are retained, breaking offline delivery guarantees.
Step 4
Encryption and Security
How will you ensure end-to-end encryption for messages stored and transmitted via Kafka?
💡 Hint
Consider Kafka’s native encryption features and application-level encryption for end-to-end guarantees.
Key Decision
Use Kafka TLS encryption for in-transit data and enable Kafka’s broker-side encryption at rest with AWS KMS integration; additionally, implement application-level encryption for true end-to-end security.
Interviewer Expectation
Mid-level: Mentions TLS and broker encryption. Senior: Explains why application-level encryption is needed for end-to-end guarantees beyond Kafka’s transport and storage encryption.
⚠️Common Wrong Answer
Relying solely on Kafka’s TLS and broker encryption - does not protect message content from Kafka admins or compromised brokers.
Step 5
API Design for Messaging
Design three core API endpoints: send message, sync messages after offline, and acknowledge message receipt. What are their inputs, outputs, and key considerations?
💡 Hint
Think about how clients specify partition keys, offsets, and acknowledgments to maintain ordering and exactly-once semantics.
Key Decision
SendMessage API accepts conversation_id, message payload, and metadata; SyncMessages API accepts conversation_id and last known offset to fetch missed messages; AcknowledgeMessage API commits offset to Kafka to update consumer progress.
Interviewer Expectation
Mid-level: Defines basic API signatures. Senior: Details how APIs interact with Kafka partitions, offset management, and handle retries and idempotency.
⚠️Common Wrong Answer
Designing a polling-based sync API without offsets - leads to inefficient data transfer and inability to guarantee message ordering or exactly-once delivery.
🔌
API Design Challenge
Sketch 3 core API endpoints for the messaging system: sending a message, syncing messages after offline period, and acknowledging message receipt.
Hints
  • How does the client send messages to ensure ordering and partitioning?
  • How does the client efficiently sync missed messages after reconnecting?
  • How does the client acknowledge messages to update consumer offsets?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of conversation_id as Kafka partition key to guarantee ordering
  • Manual offset commit in Kafka consumer groups for exactly-once processing
  • Retention policy configured with time-based limits to support offline delivery
  • TLS and broker-side encryption integrated with KMS for data security
  • API design that leverages Kafka partitioning and offset semantics
Good to Have - 30%
  • Application-level encryption for end-to-end confidentiality
  • Explanation of trade-offs between retention time and storage cost
  • Handling of consumer crashes and offset commit failures
  • Discussion of scaling Kafka clusters to handle 1M msg/sec throughput
Bonus - 10%
  • Design considerations for multi-region Kafka replication
  • Strategies for monitoring and alerting on consumer lag and retention breaches
Anti-patterns to Avoid
✅
Model Answer
▼ Reveal

The system uses Kafka with conversation_id as the partition key to ensure message ordering per conversation. Consumer groups use manual offset commits stored in Kafka's __consumer_offsets topic to support exactly-once processing and offline message delivery. Retention policies are time-based with a default of 7 days to balance storage and user experience. Security is ensured via Kafka TLS encryption in transit, broker-side encryption at rest integrated with AWS KMS, and application-level encryption for true end-to-end confidentiality. The API design includes SendMessage, SyncMessages, and AcknowledgeMessage endpoints that leverage Kafka's partitioning and offset semantics to maintain ordering and exactly-once delivery.

Key Decisions
  • Use conversation_id as Kafka partition key to guarantee ordering and consumer affinity
  • Implement manual offset commits in Kafka consumer groups stored in __consumer_offsets for exactly-once processing and offline delivery
  • Configure retention policy with time-based retention of 7 days and size limits per partition to balance storage cost and offline message availability
  • Enable Kafka TLS encryption for data in transit and broker-side encryption at rest with AWS KMS integration
  • Implement application-level encryption for true end-to-end message confidentiality beyond Kafka's native encryption
  • Design SendMessage API to accept conversation_id, message payload, and metadata to ensure correct partitioning
  • Design SyncMessages API to accept conversation_id and last known offset to efficiently fetch missed messages
  • Design AcknowledgeMessage API to commit offsets to Kafka to update consumer progress and maintain exactly-once semantics
  • conversation_id as Kafka partition key - ensures ordering and consumer affinity per conversation
  • Kafka consumer groups with manual offset commits stored in __consumer_offsets for exactly-once processing
  • Time-based retention policy of 7 days per topic balancing storage cost and offline message availability
  • Kafka TLS encryption in transit and broker-side encryption at rest integrated with AWS KMS for security
  • Application-level encryption for true end-to-end message confidentiality beyond Kafka’s native encryption
  • SendMessage, SyncMessages, and AcknowledgeMessage APIs designed to leverage Kafka partitioning and offset semantics
  • Avoid log compaction for chat topics to prevent message loss and preserve full message history for offline delivery
  • Manual offset commit strategy to prevent message loss or duplication during consumer crashes or offline periods

Practice

(1/5)
1. A distributed key-value store must evenly distribute keys across nodes and handle node failures without massive data reshuffling. Which component primarily enables this?
easy
A. Gossip protocol to propagate membership and failure information
B. Vector clocks to resolve conflicting versions of data
C. Consistent hashing to assign keys to nodes with minimal remapping
D. Quorum-based replication to ensure data durability

Solution

  1. Step 1: Identify the role of consistent hashing

    Consistent hashing distributes keys to nodes such that when nodes join or leave, only a small subset of keys need reassignment, minimizing reshuffling.

  2. Step 2: Eliminate other components

    Vector clocks handle version conflicts but do not distribute keys. Gossip protocol manages cluster membership but not key distribution. Quorum replication ensures durability but does not affect key assignment.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Consistent hashing minimizes key remapping on node changes [check]
Hint: Consistent hashing minimizes key remapping on node changes [OK]
Common Mistakes:
2. If the artifact registry becomes unavailable during deployment, what is the most likely failure mode in the CI/CD pipeline?
medium
A. Deployment proceeds with cached artifacts, risking inconsistent versions
B. Deployment triggers rollback immediately without attempting deployment
C. Deployment blocks waiting for artifact registry recovery, causing pipeline timeout
D. Deployment skips artifact validation and proceeds, risking corrupted deployments

Solution

  1. Step 1: Trace deployment behavior on artifact registry unavailability

    Deployment cannot validate or fetch artifacts, so it aborts and triggers rollback to maintain safety.

  2. Step 2: Eliminate other failure modes

    Proceeding with cached or skipping validation risks corrupted deployments; blocking indefinitely causes pipeline stalls but rollback is preferred.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Unavailable registry -> deployment aborts and triggers rollback [check]
Hint: No artifact registry -> abort deployment, rollback triggered [check]
Common Mistakes:
3. If the Progress Worker crashes before persisting a user's video watch progress, what is the exact failure observed by the user when they resume the course?
medium
A. User sees progress reset to zero and must rewatch videos from the start
B. User's progress appears stuck at last persisted checkpoint, missing recent watch time
C. User receives an error message preventing course continuation until retry
D. User's progress is duplicated causing certificate issuance errors

Solution

  1. Step 1: Trace failure impact of Progress Worker crash

    Progress Worker crash before persistence means recent watch time is lost.

  2. Step 2: Evaluate user experience on resume

    System detects missing progress update and returns error preventing continuation (User receives an error message preventing course continuation until retry), rather than silently resetting or duplicating progress.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Crash causes error blocking course continuation until retry [check]
Hint: Crash blocks progress update, user sees error [OK]
Common Mistakes:
4. For a high-traffic public API requiring both strong security and minimal latency, which TLS configuration trade-off is most appropriate?
medium
A. Use TLS 1.3 with ephemeral key exchange to enable forward secrecy and reduce handshake latency
B. Use self-signed certificates to avoid certificate chain overhead
C. Disable certificate validation to speed up connections
D. Use TLS 1.0 with RSA key exchange for compatibility

Solution

  1. Step 1: Evaluate TLS versions and key exchange

    TLS 1.3 supports ephemeral key exchange (ECDHE) which provides forward secrecy and reduces handshake round trips, lowering latency.

  2. Step 2: Eliminate other options

    Use TLS 1.0 with RSA key exchange for compatibility uses outdated TLS 1.0 and RSA key exchange lacking forward secrecy and higher latency. Disable certificate validation to speed up connections compromises security by disabling validation. Use self-signed certificates to avoid certificate chain overhead risks trust issues and is unsuitable for public APIs.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    TLS 1.3 ephemeral keys -> forward secrecy + lower latency [check]
Hint: TLS 1.3 ephemeral keys = forward secrecy + low latency [check]
Common Mistakes:
5. If the sequence number state is lost due to a node crash in a Snowflake-like ID generator, what is the immediate failure observed in the system?
medium
A. The generator blocks until the sequence number is recovered from persistent storage
B. The generator resets the timestamp to an earlier value to avoid duplicates
C. The generator produces duplicate IDs for the same timestamp causing collisions downstream
D. The generator switches to UUID v4 fallback to avoid duplicates

Solution

  1. Step 1: Understand failure handling in sequence number loss

    Some implementations fallback to UUID v4 to avoid duplicates when sequence state is lost.

  2. Step 2: Evaluate other options

    Blocking causes latency; duplicate IDs cause collisions; resetting timestamp breaks ordering guarantees.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Fallback to UUID v4 avoids duplicates despite losing ordering [check]
Hint: Fallback to UUID v4 avoids duplicates after sequence loss [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard