Bird
Raised Fist0

Data Replication Strategies - Synchronous vs Asynchronous, Quorum Writes

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Data Replication Strategies - Synchronous vs Asynchronous, Quorum Writes
45 min · senior
senior

You are designing the data replication layer for a 50 million daily active user (DAU) social media platform. The system must replicate user-generated content across multiple geographically distributed data centers to ensure high availability and durability. The business requires end-to-end encryption of data at rest and in transit. The system must support offline delivery of updates when users reconnect. The P99 write latency must be under 200ms globally. You must choose between synchronous and asynchronous replication strategies and quorum-based writes to balance consistency, availability, and latency.

Constraints
  • End-to-end encryption of data at rest and in transit is mandatory
  • Support offline delivery with eventual consistency guarantees
  • Global P99 write latency must be under 200ms
  • Data must be replicated across at least 3 geographically distributed data centers
  • System must tolerate data center failures without data loss
  • Must support quorum writes to balance consistency and availability
📋
Guided Steps
1/5
Step 1
Choosing Replication Strategy
Should the system use synchronous or asynchronous replication? What are the trade-offs at 50M DAU with 200ms P99 latency?
💡 Hint
Consider how replication latency impacts write acknowledgement and data durability guarantees.
Key Decision
Use a hybrid approach with quorum-based synchronous replication within a local data center using Apache Cassandra's QUORUM consistency level, combined with asynchronous cross-datacenter replication via Apache Kafka Connect to balance latency and durability.
Interviewer Expectation
Mid-level: Names synchronous vs asynchronous replication and basic trade-offs. Senior: Explains Cassandra QUORUM writes reduce latency by limiting replicas needed for ack, and Kafka Connect asynchronously replicates to remote DCs to avoid global latency spikes.
⚠️Common Wrong Answer
Choosing fully synchronous cross-datacenter replication causing write latency to exceed 200ms P99 and reducing availability during network partitions.
Step 2
Ensuring Data Consistency with Quorum Writes
How do quorum writes help balance consistency and availability? What quorum sizes would you choose for replication factor 3?
💡 Hint
Recall the formula W + R > N for strong consistency and how it affects latency and availability.
Key Decision
Set write quorum W=2 and read quorum R=2 for replication factor N=3 in Cassandra to ensure strong consistency with minimal latency impact and tolerate one replica failure.
Interviewer Expectation
Mid-level: States quorum formula W+R>N. Senior: Explains why W=2 reduces write latency compared to full replication, how R=2 ensures read consistency, and trade-offs in availability during failures.
⚠️Common Wrong Answer
Using full replication writes (W=3) causing high write latency and reduced availability during replica failures.
Step 3
Handling Offline Delivery and Update Synchronization
How does the system support offline clients syncing missed updates efficiently and securely?
💡 Hint
Think about using change data capture and client-side cursors or vector clocks for incremental sync.
Key Decision
Use Kafka topics with partition key as user_id for ordered change streams, combined with client-maintained offsets to fetch incremental updates securely after offline periods.
Interviewer Expectation
Mid-level: Mentions Kafka or CDC for updates. Senior: Details partitioning by user_id to guarantee ordering, offset management for efficient sync, and encryption of data in Kafka topics.
⚠️Common Wrong Answer
Polling entire dataset or using HTTP polling causing excessive load and latency spikes at 50M DAU scale.
Step 4
Encrypting Data at Rest and In Transit
How do you ensure end-to-end encryption for replicated data across data centers without compromising performance?
💡 Hint
Consider TLS for in-transit and transparent encryption for storage layers, plus key management.
Key Decision
Enable TLS 1.3 for all inter-service communication including Kafka and Cassandra clients, and use Cassandra Transparent Data Encryption (TDE) with AWS KMS for key management to secure data at rest.
Interviewer Expectation
Mid-level: Mentions TLS and encryption at rest. Senior: Explains TLS 1.3 benefits for low latency, TDE integration with KMS for key rotation, and impact on write latency.
⚠️Common Wrong Answer
Encrypting data at application layer only causing complex key management and increased latency.
Step 5
API Design for Replication System
Design 3 core API endpoints to handle client writes, replication status queries, and offline update sync.
💡 Hint
Think about write acknowledgement, consistency level query, and incremental update fetch with cursors.
Key Decision
POST /write with payload and consistency param returns write ack after quorum; GET /replication-status returns replication lag and consistency info; GET /sync-updates?offset= returns ordered updates since offset for offline sync.
Interviewer Expectation
Mid-level: Defines basic endpoints. Senior: Details parameters for consistency levels, offset management, encryption headers, and error handling for partial replication failures.
⚠️Common Wrong Answer
Single endpoint for all operations causing complexity and poor client experience.
🔌
API Design Challenge
Sketch 3 core API endpoints for the replication system that handle client writes, replication status queries, and offline update synchronization.
Hints
  • How does the client submit a write and receive acknowledgement?
  • How can clients query replication status or consistency guarantees?
  • How does the system enable clients to sync missed updates after offline periods?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Clear explanation of synchronous vs asynchronous replication trade-offs at 50M DAU scale
  • Use of quorum writes with specific W and R values for consistency and availability
  • Mechanism for offline update synchronization using ordered change streams and offsets
  • End-to-end encryption strategy covering both in-transit and at-rest data
Good to Have - 30%
  • Use of Apache Cassandra and Kafka with partition keys aligned to user or conversation IDs
  • API design with separate endpoints for writes, status, and sync with clear parameters
  • Discussion of failure scenarios and how quorum writes mitigate data loss
  • Consideration of key management and encryption performance impact
Bonus - 10%
  • Detailed failure mode analysis tracing impact on latency and availability
Anti-patterns to Avoid
✅
Model Answer
▼ Reveal

The system uses a hybrid replication strategy combining synchronous quorum writes within local data centers using Apache Cassandra and asynchronous cross-datacenter replication via Apache Kafka Connect. This balances strong consistency, low latency, and high availability for 50M DAU with strict 200ms P99 write latency. Offline clients sync missed updates efficiently using Kafka topics partitioned by user_id with client-maintained offsets. End-to-end encryption is ensured with TLS 1.3 for in-transit data and Cassandra Transparent Data Encryption integrated with AWS KMS for at-rest data.

Key Decisions
  • Use Cassandra QUORUM writes with W=2 and R=2 for replication factor N=3 to balance consistency and latency
  • Employ asynchronous cross-datacenter replication with Kafka Connect to avoid global write latency spikes
  • Partition Kafka topics by user_id to guarantee ordered updates for offline sync
  • Clients maintain offsets to fetch incremental updates efficiently after offline periods
  • Enable TLS 1.3 for low-latency encryption in transit and use TDE with AWS KMS for secure key management
  • Apache Cassandra with QUORUM consistency for local synchronous replication to meet 200ms P99 latency
  • Apache Kafka Connect for asynchronous cross-datacenter replication to avoid global write latency spikes
  • Kafka partition key as user_id for ordered change streams enabling efficient offline sync
  • TLS 1.3 for all inter-service communication ensuring low-latency encryption in transit
  • Cassandra Transparent Data Encryption (TDE) integrated with AWS KMS for secure key management and data at rest encryption
  • Write quorum W=2 and read quorum R=2 for replication factor N=3 balancing consistency and availability
  • Client-maintained offsets for incremental update synchronization after offline periods
  • API endpoints designed with explicit consistency parameters and offset-based sync for clarity and performance

Practice

(1/5)
1. A REST API needs to support clients fetching large lists of user records without overwhelming the server or client memory. Which design principle best addresses this requirement?
easy
A. Implement pagination with query parameters like page and limit
B. Use HTTP POST to send the entire user list in the request body
C. Return all user records in a single response with HTTP 200 status
D. Use HTTP PUT to update the user list incrementally

Solution

  1. Step 1: Identify the need to limit response size

    Fetching large lists risks high memory usage and slow responses; pagination limits data per request.

  2. Step 2: Recognize pagination as the standard REST approach

    Using query parameters like page and limit allows clients to request manageable chunks.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Pagination with query params controls response size [check]
Hint: Paginate large lists with query params [check]
Common Mistakes:
2. Without the consistent hashing ring metadata, what exactly breaks in a distributed cache system when a client requests a key?
medium
A. The cache nodes lose synchronization and data becomes corrupted
B. The cache node receiving the request cannot determine the correct node, causing request failure or indefinite retry
C. The client receives stale data from an outdated cache node
D. The system falls back to a centralized database for all requests, increasing latency

Solution

  1. Step 1: Role of consistent hashing ring metadata

    The ring metadata maps keys to nodes; without it, nodes cannot route requests correctly, leading to inconsistent data placement.

  2. Step 2: Consequence of missing ring metadata

    Without this metadata, nodes lose synchronization on key ownership, causing data corruption as writes go to wrong nodes; request failure (The cache node receiving the request cannot determine the correct node, causing request failure or indefinite retry) is secondary, and fallback (The system falls back to a centralized database for all requests, increasing latency) is not automatic.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Missing ring metadata causes data corruption due to lost key-node mapping [check]
Hint: No ring metadata -> lost key-node mapping -> data corruption [check]
Common Mistakes:
3. For propagating trace context across asynchronous messaging systems, which approach best preserves trace continuity without coupling services tightly?
medium
A. Embed trace context in message metadata headers supported by the messaging system
B. Use HTTP headers to propagate trace context in message payloads directly
C. Store trace context in a centralized database and have consumers query it on message receipt
D. Generate new trace IDs at each service and correlate offline during analysis

Solution

  1. Step 1: Understand asynchronous trace context propagation

    Embedding trace context in message metadata headers leverages messaging system features to propagate context without modifying payload or tight coupling.

  2. Step 2: Evaluate other options

    HTTP headers don't exist in messaging protocols. Centralized DB lookup adds latency and coupling. Generating new trace IDs breaks trace continuity and requires complex offline correlation.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Message metadata headers preserve trace context asynchronously [check]
Hint: Use message metadata headers, not payload or DB lookup [OK]
Common Mistakes:
4. If the client fails to release the lock on one Redis master due to network partition, what is the exact failure mode observed in the Redlock algorithm?
medium
A. The lock remains held on that master until expiration, causing potential deadlock
B. The client can reacquire the lock instantly without waiting
C. The lock is immediately available to other clients on that master
D. The lock is automatically transferred to another client

Solution

  1. Step 1: Understand lock release in Redlock

    Locks are released by explicit DEL commands; network partition can prevent this.

  2. Step 2: Trace failure impact

    If lock not released, it remains held until TTL expires, potentially blocking other clients.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Unreleased lock -> held until TTL expires, causing potential deadlock [check]
Hint: Unreleased lock -> held until TTL expires, blocking others [check]
Common Mistakes:
5. At 10x scale, which component in a range-sharded database system is most likely to fail first and why?
hard
A. The shard directory service, because it becomes a centralized bottleneck for routing requests
B. The shard servers holding the highest traffic ranges, due to uneven load and hotspotting
C. The network load balancer, because it cannot handle the increased number of shards
D. The client-side hashing logic, because it cannot scale to map keys to shards

Solution

  1. Step 1: Identify load distribution in range sharding

    Range sharding partitions data by key ranges, causing uneven load if some ranges are hotter.

  2. Step 2: Analyze scaling impact

    At 10x scale, shards with hot ranges become overloaded first, causing failures before other components.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Range shard hotspots cause uneven load and early failures [check]
Hint: Range shards fail at hotspots due to uneven load [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard