Bird
Raised Fist0

Zookeeper for Leader Election and Distributed Configuration

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Zookeeper for Leader Election and Distributed Configuration
Design a highly available, consistent coordination service for leader election and distributed config management using Zookeeper
mediumDistributed CoordinationFreq: common
⚙️
Functional Requirements
  • Provide a reliable leader election mechanism for distributed services to coordinate tasks.
  • Store and manage distributed configuration data with strong consistency guarantees.
  • Support ephemeral nodes to track live session presence and enable automatic cleanup on client disconnect.
  • Allow clients to watch for changes on znodes and receive notifications for updates.
  • Ensure atomic updates and versioning of configuration data to prevent race conditions.
  • Support hierarchical namespace for configuration keys and ephemeral nodes.
📊
Non-Functional Requirements
  • Availability: 99.99% uptime to avoid service disruption in leader election and config reads.
  • Latency: 100ms P99 read latency for config fetch and watch notifications.
  • Throughput: Support up to 1 million write QPS for ephemeral node updates and config changes.
  • Consistency: Linearizable consistency for all read/write operations to prevent split-brain.
  • Scalability: Handle up to 10,000 concurrent clients with session management.
❓
Clarifying Questions to Ask
  • What is the expected number of concurrent clients and session churn rate?
  • What is the maximum size of configuration data stored per znode?
  • Are ephemeral nodes required to expire immediately on client disconnect?
  • Is strong linearizable consistency mandatory or is eventual consistency acceptable?
  • What is the expected read-to-write ratio for configuration data?
  • Should the system support hierarchical watches or only single-node watches?
Out of Scope
✗ Designing the client-side retry and backoff logic for session reconnects.
✗ Implementing the application logic that uses leader election results.
✗ Data analytics or long-term storage of configuration change history.
✗ Multi-datacenter geo-replication and disaster recovery.
📌
Assumptions
  • 10,000 concurrent clients with average session churn of 1 session per minute.
  • Each client sends 40 ephemeral node updates per day (heartbeat + ephemeral node create/delete).
  • Average configuration write size is 100 bytes per znode update.
  • Read QPS is 10x write QPS due to frequent config fetches and watch notifications.
⚡
QPS Calculation
write qps10,000 clients × 40 updates/day ÷ 86400 sec ≈ 4.6 writes/sec
peak write qpsAssuming 10× peak factor: 4.6 × 10 = 46 writes/sec
read qps10× write QPS: 46 × 10 = 460 reads/sec
💾
Storage Estimate
daily storage4.6 writes/sec × 86400 sec × 100 bytes = 39.7 MB/day
retention days7
total storage39.7 MB/day × 7 days = 278 MB
with replication3 replicas × 278 MB = 834 MB total storage
🖥️
Server Estimate
zookeeper servers3-5 servers for quorum to tolerate 1-2 failures and maintain consistency
client proxies2 servers to handle client connections and session management at 10,000 clients
🏗️
Architecture Diagram
Architecture Diagram
          +----------------------+
          |  Clients (10,000)    |
          +----------+-----------+
                     |
          +----------v-----------+
          |  Client Proxies /    |
          |  Session Managers    |
          +----------+-----------+
                     |
          +----------v-----------+
          |  Zookeeper Quorum    |
          |  (3-5 Servers)       |
          +----------+-----------+
                     |
          +----------v-----------+
          |  Persistent Storage  |
          |  (Write-Ahead Log +  |
          |   Snapshot Storage)  |
          +----------------------+
🧩
Components
Zookeeper Quorum
Maintain a replicated, consistent state machine for leader election and config storage
Why thisZookeeper uses Zab protocol for atomic broadcast ensuring linearizable consistency and fault tolerance
Why notEtcd uses Raft but Zab provides better write throughput for leader election workloads
Client Proxies / Session Managers
Manage client sessions, ephemeral nodes, and watch registrations to offload quorum servers
Why thisSeparates session state from quorum to reduce load and improve scalability for 10,000 clients
Why notDirect client connections to quorum overload servers and increase latency under high churn
Persistent Storage (WAL + Snapshots)
Durably store transaction logs and periodic snapshots for recovery and state reconstruction
Why thisWrite-Ahead Log ensures durability and snapshots speed up recovery time after failures
Why notIn-memory only storage risks data loss on server crashes; traditional RDBMS lacks linearizable write throughput
Ephemeral Node Mechanism
Track client liveness and automatically delete ephemeral znodes on session expiration
Why thisEnables reliable leader election and presence detection without manual cleanup
Why notTTL-based caches like Redis do not guarantee immediate cleanup on client disconnect
Watch Notification System
Notify clients asynchronously on znode data or child changes to enable reactive config updates
Why thisPush-based notifications reduce client polling and improve latency for config changes
Why notPolling-based systems increase load and latency, unsuitable for 100ms P99 latency
🔄
Data Flow
1
Client → Client Proxy
Client opens session and registers ephemeral nodes
Session manager tracks client liveness and ephemeral node lifecycle
2
Client Proxy → Zookeeper Quorum
Proxy forwards write requests (leader election, config updates) to quorum
Quorum orders writes via Zab protocol ensuring linearizability
3
Zookeeper Quorum → Persistent Storage
Quorum logs transaction to Write-Ahead Log and applies to in-memory state
Durability and crash recovery ensured by WAL and snapshots
4
Zookeeper Quorum → Client Proxy
Quorum confirms write success and triggers watch notifications
Client proxies notify subscribed clients asynchronously
5
Client Proxy → Client
Proxy sends watch event notifications or read responses
Clients react to config changes or leader election results
6
Client → Client Proxy
Client sends heartbeats to maintain session and ephemeral nodes
Session manager expires ephemeral nodes on heartbeat timeout
🗄️
znodes
PK: path
path STRINGdata BLOBversion INTephemeral BOOLEANowner_session_id STRINGcreated_at TIMESTAMPmodified_at TIMESTAMP
🗄️
sessions
PK: session_id
session_id STRINGclient_id STRINGlast_heartbeat TIMESTAMPstate STRING
🔌
CreateSession
Request
{
  "client_id": "STRING"
}
Response
{
  "session_id": "STRING",
  "expires_in": "INT"
}
Client opens a session to start ephemeral node lifecycle
🔌
CreateZnode
Request
{
  "session_id": "STRING",
  "path": "STRING",
  "data": "BLOB",
  "ephemeral": "BOOLEAN"
}
Response
{
  "success": "BOOLEAN",
  "version": "INT"
}
Create a new znode, optionally ephemeral tied to session
🔌
WatchZnode
Request
{
  "session_id": "STRING",
  "path": "STRING"
}
Response
{
  "watch_id": "STRING"
}
Register a watch on a znode to receive change notifications

Practice

(1/5)
1. In the payment gateway request flow, at Step 4 the Webhook Delivery Worker attempts to send a payment status update to a merchant's endpoint. If the merchant's server responds with HTTP 500, what does the Delivery Worker do next?
easy
A. Marks the webhook as delivered and stops retrying to avoid duplicate notifications
B. Retries the webhook delivery asynchronously with exponential backoff and jitter
C. Immediately retries the webhook delivery synchronously until success or timeout
D. Drops the webhook notification and logs an error for manual intervention

Solution

  1. Step 1: Understand webhook failure handling

    Transient failures like HTTP 500 require retries to ensure eventual delivery without blocking the main flow.

  2. Step 2: Evaluate retry strategy

    Asynchronous retries with exponential backoff and jitter prevent overload and improve success chances.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Webhook retries use async exponential backoff with jitter [check]
Hint: Retry webhooks asynchronously with backoff and jitter [check]
Common Mistakes:
2. A system needs to estimate the number of unique users visiting a website daily, with minimal memory usage and acceptable error margin. Which component is best suited to handle this requirement?
easy
A. A Redis cache storing all user IDs explicitly
B. A relational database with unique user ID indexing
C. A HyperLogLog data structure for cardinality estimation
D. A Bloom filter to check if a user has visited before

Solution

  1. Step 1: Understand memory constraints and error tolerance

    Storing all user IDs explicitly (Redis or relational DB) is memory-heavy and not scalable for large user bases.

  2. Step 2: Identify HyperLogLog's role

    HyperLogLog provides approximate unique counts with fixed small memory and tunable error, ideal for large-scale cardinality estimation.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    HyperLogLog trades accuracy for memory efficiency [check]
Hint: Approximate unique counts with tiny memory -> HyperLogLog [check]
Common Mistakes:
3. A production system needs to continuously measure its reliability against defined error budgets and alert when the budget is close to exhaustion. Which component is primarily responsible for aggregating error rates and calculating remaining error budget in real time?
easy
A. A distributed log like Kafka that stores raw event data for offline batch processing
B. A metrics aggregation service that computes SLO compliance and error budget burn rates
C. A tracing system that collects detailed request traces for root cause analysis
D. A dashboard visualization tool that displays historical uptime percentages

Solution

  1. Step 1: Identify the component responsible for real-time error budget calculation

    The metrics aggregation service continuously processes incoming metrics to compute SLO compliance and error budget burn rates, enabling timely alerts.

  2. Step 2: Eliminate other options based on their roles

    Kafka stores raw data but does not compute error budgets in real time; tracing systems focus on request-level diagnostics, not aggregate error rates; dashboards visualize data but do not perform calculations.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Metrics aggregation service -> real-time error budget tracking [check]
Hint: Metrics aggregation calculates error budgets in real time [OK]
Common Mistakes:
4. If the metadata server in a file storage system crashes, what exactly breaks in the file write operation?
medium
A. File data blocks are lost permanently because they were never written to disk.
B. The system automatically switches to object storage to handle writes.
C. Clients can continue writing files normally without any impact.
D. Clients can write data blocks, but file metadata updates are lost, causing inconsistent directory state.

Solution

  1. Step 1: Understand metadata server role in file writes

    The metadata server manages file metadata; if it crashes, metadata updates are lost but data blocks may be written.

  2. Step 2: Trace failure impact

    Without metadata updates, directory entries and inode info become inconsistent, leading to orphaned or inaccessible files despite data blocks existing.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Metadata server crash -> lost metadata, inconsistent directory [check]
Hint: Metadata server down -> metadata lost, data blocks orphaned [check]
Common Mistakes:
5. At 10x transaction volume, which component in the fraud detection system is most likely to become the first bottleneck, and why?
hard
A. Rule Engine, because complex rules increase CPU usage linearly with transactions and cannot be parallelized
B. Feature Store, because it must serve low-latency queries for all transactions and cannot scale horizontally easily
C. Kafka ingestion cluster, because partition count limits parallelism and increasing partitions impacts ordering guarantees
D. Real-Time Scoring service, because it maintains stateful caches per user that cannot be sharded easily

Solution

  1. Step 1: Analyze Feature Store scaling (A)

    Feature Stores often scale horizontally with caching and sharding; not usually first bottleneck.

  2. Step 2: Analyze Kafka ingestion cluster (B)

    Kafka partitions limit parallelism; increasing partitions can break ordering guarantees per user, making scaling challenging and causing ingestion bottlenecks.

  3. Step 3: Analyze Rule Engine and Scoring (C and D)

    Rule Engine can be scaled horizontally with stateless evaluation; scoring caches can be sharded by user ID, so less likely first bottleneck.

  4. Final Answer:

    Option C -> Option C

  5. Quick Check:

    Kafka partition limits -> ingestion bottleneck at 10x scale [check]
Hint: Kafka partitions limit ingestion scale before others [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard