Bird
Raised Fist0

Kafka Architecture - Partitions, Consumer Groups, Offsets, Retention Policy

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Kafka Architecture - Partitions, Consumer Groups, Offsets, Retention Policy
Designing Kafka's core architecture focusing on partitions, consumer groups, offsets, and retention policies at scale
mediumMessage Queues and Event StreamingFreq: high
Prerequisites:Message QueuesPub-Sub Pattern
⚙️
Functional Requirements
  • Support high-throughput message ingestion with partitioned topics for parallelism
  • Enable multiple consumer groups to independently consume messages from topics
  • Track consumer offsets to allow exactly-once or at-least-once processing semantics
  • Implement configurable retention policies to manage storage and data lifecycle
  • Ensure message ordering within partitions
  • Support fault tolerance and recovery for producers and consumers
📊
Non-Functional Requirements
  • Handle 2 billion daily active users producing 40 messages per user per day
  • Provide 99.99% availability for message ingestion and consumption
  • Maintain end-to-end latency under 100ms P99 for message delivery
  • Support peak write throughput of approximately 925,000 messages per second
  • Ensure data durability with replication factor of 3
❓
Clarifying Questions to Ask
  • What is the expected message size and variability?
  • Are messages strictly ordered across the entire topic or only within partitions?
  • What consistency guarantees are required for offset commits?
  • What is the maximum retention period for messages?
  • How many consumer groups and consumers per group are expected?
  • Is the system expected to support transactional message processing?
Out of Scope
✗ Design of Kafka Connect or Kafka Streams components
✗ Security features such as encryption and authentication
✗ Detailed producer or consumer client SDK implementations
✗ Cross-datacenter replication and geo-replication
📌
Assumptions
  • 2 billion daily active users (DAU)
  • Each user produces 40 messages per day
  • Average message size is 100 bytes
  • Replication factor is 3 for durability
⚡
QPS Calculation
write qps2,000,000,000 × 40 / 86,400 ≈ 925,926 msg/sec
peak write qpsAssuming 2× peak factor: 925,926 × 2 ≈ 1,851,852 msg/sec
read qpsAssuming 1.5× read factor due to multiple consumers: 925,926 × 1.5 ≈ 1,388,889 msg/sec
💾
Storage Estimate
daily storage2,000,000,000 × 40 × 100 bytes = 8,000,000,000,000 bytes = 8 TB/day
retention days7
total storage8 TB × 7 = 56 TB
with replication56 TB × 3 = 168 TB total storage required
🖥️
Server Estimate
broker serversAssuming each Kafka broker handles 20,000 msg/sec write throughput, need 1,851,852 / 20,000 ≈ 93 brokers
zookeeper servers3-5 servers for quorum and metadata management
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          |   Producers    |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |   Kafka Broker |
          | (Partitioned   |
          |   Topics)      |
          +-------+--------+
                  |
                  v
          +-------+--------+
          | Consumer Group |
          |  Consumers     |
          +-------+--------+
                  |
                  v
          +-------+--------+
          |  Offset Store  |
          | (Kafka __consumer_offsets topic) |
          +----------------+
🧩
Components
Kafka Broker
Stores topic partitions, manages message writes, replication, and retention
Why thisKafka brokers efficiently handle high-throughput partitioned logs with zero-copy and segment files for fast disk IO
Why notRabbitMQ uses queue-based routing which limits partition parallelism and scales poorly beyond 100K msg/sec
Partitions
Divide topics into ordered, immutable sequences to enable parallelism and scalability
Why thisPartitions allow horizontal scaling by distributing load across brokers and preserving message order within partitions
Why notSingle log per topic limits throughput and causes bottlenecks at scale
Consumer Groups
Allow multiple consumers to coordinate consumption of partitions for load balancing and fault tolerance
Why thisConsumer groups enable parallel consumption with offset tracking, supporting scalable and independent processing
Why notDirect consumer subscriptions without groups cause duplicate processing and lack fault tolerance
Offset Store (__consumer_offsets topic)
Persist consumer offsets in a compacted Kafka topic for fault-tolerant offset management
Why thisUsing Kafka itself for offset storage ensures consistency, durability, and scalability without external dependencies
Why notZookeeper is deprecated for offset storage due to poor scalability and high latency
Retention Policy
Controls how long messages are retained on brokers before deletion based on time or size
Why thisConfigurable retention allows balancing storage costs and data availability for replay or late consumers
Why notManual deletion or infinite retention leads to unbounded storage growth and operational complexity
🔄
Data Flow
1
Producer → Kafka Broker
Producer sends message to topic partition based on partition key
Partition key ensures message ordering within partition
2
Kafka Broker → Broker Storage
Broker appends message to partition log segment on disk
Zero-copy and segment files optimize disk IO
3
Kafka Broker → Replica Brokers
Replicate message to follower brokers asynchronously
Replication factor 3 ensures durability and fault tolerance
4
Consumer Group → Kafka Broker
Consumers fetch messages from assigned partitions
Consumers coordinate partition assignment within group
5
Consumer → Offset Store
Consumer commits offset after processing message
Offset commit stored in __consumer_offsets topic for recovery
6
Kafka Broker → Retention Manager
Periodically delete messages older than retention policy
Prevents unbounded storage growth

Practice

(1/5)
1. A web application experiences increasing CPU load on a single database server, causing query latency spikes. Which scaling approach best addresses this bottleneck without changing the application code?
easy
A. Distribute the database load by sharding data across multiple servers (horizontal scaling)
B. Add more CPU cores and RAM to the existing database server (vertical scaling)
C. Use a caching layer in front of the database to reduce read load
D. Switch from SQL to NoSQL database to improve write throughput

Solution

  1. Step 1: Identify the bottleneck as CPU and memory on a single DB server

    Vertical scaling adds resources to the existing server to handle more load without architectural changes.

  2. Step 2: Evaluate horizontal scaling and other options

    Horizontal scaling requires application changes for sharding; caching reduces reads but not CPU-bound writes; switching DB types is a different trade-off.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Vertical scaling adds resources to one server without code changes [check]
Hint: Vertical scaling adds power to one machine [check]
Common Mistakes:
2. In the unique ID generation system, at step 3 the ID generator node receives a timestamp and node ID. What state does the generator maintain to ensure uniqueness when the timestamp is the same as the previous request?
easy
A. It resets the sequence number to zero for each new timestamp
B. It increments a sequence number to differentiate IDs within the same timestamp
C. It waits for the timestamp to advance before generating the next ID
D. It generates a random suffix to append to the ID

Solution

  1. Step 1: Understand sequence number reset behavior

    For each new timestamp, the sequence number resets to zero to start fresh counting.

  2. Step 2: Ensure uniqueness within timestamp by incrementing sequence

    Within the same timestamp, the sequence number increments; resetting happens only when timestamp changes.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Sequence number resets on new timestamp -> uniqueness maintained [check]
Hint: Sequence number resets to zero on new timestamp [check]
Common Mistakes:
3. For a pub-sub system requiring low latency fan-out to millions of subscribers with dynamic topic subscriptions, which architecture is best suited?
medium
A. Use a centralized broker that pushes messages directly to all subscribers synchronously
B. Use a multicast network protocol to broadcast messages to all subscribers
C. Use a polling-based system where subscribers fetch messages periodically from a database
D. Use a distributed log system with topic partitions and consumer groups for parallel consumption

Solution

  1. Step 1: Evaluate synchronous push and multicast

    Centralized synchronous push (Use a centralized broker that pushes messages directly to all subscribers synchronously) causes bottlenecks and backpressure. Multicast (Use a multicast network protocol to broadcast messages to all subscribers) lacks reliability and fine-grained subscription control.

  2. Step 2: Assess polling and distributed log

    Polling (Use a polling-based system where subscribers fetch messages periodically from a database) adds latency and load. Distributed log with partitions and consumer groups (Use a distributed log system with topic partitions and consumer groups for parallel consumption) supports scalable, parallel, and dynamic subscriptions with low latency.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Distributed logs scale fan-out with parallel consumers [check]
Hint: Distributed logs + consumer groups = scalable low-latency fan-out [check]
Common Mistakes:
4. Without the Rendezvous Hashing component in a distributed cache, what exactly breaks when a node fails?
medium
A. Cache keys are evenly redistributed instantly with no downtime
B. The system blocks all requests until a new mapping is manually configured
C. Clients must retry multiple nodes, causing increased latency and inconsistent cache hits
D. Requests are routed to a random node, causing cache misses and stale data

Solution

  1. Step 1: Understand role of Rendezvous Hashing in failure

    Rendezvous Hashing provides deterministic mapping without central coordination. Without it, no automatic remapping occurs.

  2. Step 2: Trace failure impact

    Without Rendezvous Hashing, system blocks or requires manual remapping, causing request blocking and downtime.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    No automatic remapping -> system blocks until manual config [check]
Hint: No Rendezvous -> manual remap blocks requests [check]
Common Mistakes:
5. At 10x scale, which component is most likely to fail first and why in Facebook Messenger's group chat system?
hard
A. Chat Servers because WebSocket connections are stateful and cannot be resharded easily
B. The database because it cannot handle 10x write throughput
C. Kafka brokers because partition count cannot be increased dynamically
D. Presence service because Redis TTL expiration causes message loss

Solution

  1. Step 1: Analyze stateful components at 10x scale

    Chat Servers maintain WebSocket connections bound to server memory, limiting horizontal scaling.

  2. Step 2: Compare other components

    Databases and Kafka can be sharded or scaled; Redis TTL issues cause transient problems but not first failure.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Stateful WebSocket connections -> Chat Servers bottleneck at scale [check]
Hint: Stateful WebSocket binds connections -> Chat Servers bottleneck [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard