Bird
Raised Fist0

Kafka Architecture - Partitions, Consumer Groups, Offsets, Retention Policy

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/5
Kafka Broker

Receive message from producer

Producer sends message to Kafka Broker for a topic

Protocol:Kafka Produce
BeforeMessage exists only on producer side
AfterMessage accepted by Kafka Broker for partitioning
Partitions

Store message in partition

Message appended to the correct topic partition log

Protocol:Internal Kafka storage
BeforePartition log has previous messages up to offset N
AfterPartition log includes new message at offset N+1
Consumer Groups

Fetch message from partition

Consumer group member reads message from partition offset

Protocol:Kafka Fetch
BeforeConsumer offset is at N
AfterConsumer reads message at offset N+1
Offset Store (__consumer_offsets topic)

Commit offset after processing

Consumer commits offset N+1 to offset store

Protocol:Kafka Commit Offset
BeforeCommitted offset is N
AfterCommitted offset updated to N+1
Retention Policy

Delete old messages

Retention policy deletes messages older than configured time

Protocol:Internal Kafka cleanup
BeforePartition contains messages older than retention period
AfterOld messages removed from partition storage
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────────┐
│   Kafka Broker    │
└─────────┬─────────┘
          │ Kafka Produce
          ▼
┌───────────────────┐
│    Partitions     │
│ (Topic partitions)│
└─────────┬─────────┘
          │ Kafka Fetch
          ▼
┌───────────────────┐
│ Consumer Groups   │
│ (Consumers read)  │
└─────────┬─────────┘
          │ Kafka Commit Offset
          ▼
┌───────────────────────────────┐
│ Offset Store (__consumer_offsets)│
│ (Stores committed offsets)    │
└─────────┬─────────────────────┘
          │ Enforces retention
          ▼
┌───────────────────┐
│ Retention Policy  │
│ (Deletes old data)│
└───────────────────┘
Design Rationale

This Kafka architecture diagram shows the flow of a message from producer to consumer with key components. The Kafka Broker receives the message and appends it to the appropriate partition, ensuring ordered storage. Consumer Groups fetch messages from partitions and commit their read offsets to the Offset Store (__consumer_offsets topic), enabling fault-tolerant tracking of consumption progress. The Retention Policy component enforces data lifecycle by deleting messages older than the configured retention period, preventing unbounded storage growth. This design balances high-throughput message ingestion, ordered delivery, and consumer state management.

Primary Focus

✓

Practice

(1/5)
1. In the request flow of a full-text search system, at Step 4 the Indexing Worker reads a batch of document update events from Kafka. What happens if the Indexing Worker crashes before committing the offset?
easy
A. The batch is lost and the index misses those updates permanently
B. The offset is auto-committed, so the batch is skipped on restart
C. Kafka re-delivers the same batch after the worker restarts, causing possible duplicate indexing
D. The Indexing Worker switches to a backup Kafka partition to continue processing

Solution

  1. Step 1: Understand Kafka offset commit semantics

    If the worker crashes before committing the offset, Kafka will re-deliver the same batch after restart to ensure no data loss.

  2. Step 2: Analyze implications of re-delivery

    This can cause duplicate indexing unless the system is idempotent or deduplicates updates.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Uncommitted Kafka offsets cause re-delivery on restart [check]
Hint: Uncommitted Kafka offsets cause re-delivery on restart [check]
Common Mistakes:
2. Without a message queue like Kafka in a chat system, what exactly breaks in message delivery?
medium
A. Alice's send blocks until Bob's server responds; if Bob's server crashes, message is lost with no retry
B. Messages are duplicated due to lack of idempotency
C. Messages are encrypted incorrectly causing security breaches
D. Delivery workers lose track of message ordering causing out-of-order display

Solution

  1. Step 1: Role of Kafka in decoupling send and delivery

    Kafka buffers messages asynchronously, preventing sender blocking on recipient availability.

  2. Step 2: Consequence of removing Kafka

    Sender must synchronously wait for recipient response; if recipient crashes, message is lost with no retry.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    No queue causes synchronous blocking and message loss on recipient failure [check]
Hint: No queue means sender blocks and loses messages if recipient crashes [check]
Common Mistakes:
3. For a globally distributed key-value store requiring low write latency and strong consistency, which replication approach is best and why?
medium
A. Single-leader replication with leader in primary region only
B. Asynchronous replication with conflict resolution on reads
C. Synchronous replication with majority quorum writes to all regions
D. Synchronous replication with local quorum writes per region and global reconciliation

Solution

  1. Step 1: Evaluate latency and consistency trade-offs

    Global synchronous quorum to all regions (A) causes high latency; asynchronous (B) sacrifices strong consistency.

  2. Step 2: Consider local quorum with global reconciliation

    Local quorum writes reduce latency while global reconciliation maintains strong consistency.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Local quorum + global reconciliation balances latency and consistency [check]
Hint: Local quorum writes reduce latency, global reconciliation ensures consistency [check]
Common Mistakes:
4. Without the Tenant Metadata Service, what exactly breaks in the multi-tenant SaaS platform's request flow?
medium
A. Tenant-specific feature flags fail to load, causing all features to be disabled globally
B. Tenant context is missing, so requests default to a shared tenant, causing data leakage
C. Billing events cannot be attributed to tenants, resulting in lost revenue tracking
D. Authentication tokens cannot be validated, blocking all tenant requests

Solution

  1. Step 1: Identify Tenant Metadata Service role

    This service provides tenant context metadata used for billing attribution and feature flag evaluation.

  2. Step 2: Trace failure impact without Tenant Metadata Service

    Without it, billing events lack tenant IDs, so usage cannot be correctly attributed, losing revenue tracking.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Missing tenant metadata -> billing events unassigned, lost revenue [check]
Hint: Tenant metadata missing -> billing attribution breaks [check]
Common Mistakes:
5. If the Kafka cluster responsible for connection event streaming goes down, what exactly breaks in the system's connection request processing?
medium
A. Connection requests are lost immediately with no retry or buffering
B. Connection requests are redirected to a backup database that duplicates all writes
C. Connection requests block the user synchronously until Kafka recovers, causing UI timeouts
D. Connection requests are accepted but downstream services do not receive updates, causing stale connection graphs

Solution

  1. Step 1: Understand Kafka's role in event streaming

    Kafka buffers and reliably delivers connection events to downstream services asynchronously.

  2. Step 2: Trace failure impact

    If Kafka is down, writes to Kafka fail but connection requests may still be accepted at the API layer; however, downstream consumers do not get updates, causing stale graph state.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Kafka down -> downstream stale, no immediate request loss [check]
Hint: Kafka failure stalls downstream, not immediate request loss [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard