Consistency models (strong, eventual) in HLD - System Design Exercise

Design: Distributed Data Store with Consistency Models

Design focuses on consistency models implementation in a distributed data store. Does not cover user interface or specific database query languages.

Functional Requirements

FR1: Support read and write operations on distributed data

FR2: Provide strong consistency option where reads always see the latest writes

FR3: Provide eventual consistency option where reads may see stale data but system converges eventually

FR4: Allow clients to choose consistency model per operation

FR5: Handle network partitions and node failures gracefully

FR6: Ensure data durability and availability

Non-Functional Requirements

NFR1: System should handle 10,000 concurrent clients

NFR2: Read latency under strong consistency: p99 < 200ms

NFR3: Read latency under eventual consistency: p99 < 50ms

NFR4: Availability target: 99.9% uptime

NFR5: Data replication across at least 3 nodes for fault tolerance

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Client API layer to specify consistency preference

Replication protocol (e.g., consensus for strong consistency)

Conflict resolution mechanism for eventual consistency

Data storage nodes

Failure detection and recovery modules

Design Patterns

Consensus algorithms (Paxos, Raft) for strong consistency

Gossip protocols for eventual consistency

Quorum reads and writes

Vector clocks or version vectors for conflict detection

Reference Architecture

          +-------------------+
          |      Clients      |
          +---------+---------+
                    |
        +-----------+-----------+
        |    API Gateway Layer   |
        | (Consistency Selector) |
        +-----------+-----------+
                    |
       +------------+------------+
       |                         |
+------+-----+             +-----+-------+
| Strong Consistency Node  |  Eventual    |
|  Cluster (Consensus)     |  Consistency |
|  (Raft/Paxos)            |  Cluster     |
+------+-----+             +-----+-------+
       |                         |
       +------------+------------+
                    |
           +--------+--------+
           |   Persistent    |
           |   Storage       |
           +-----------------+

Components

API Gateway Layer

Custom service

Receives client requests and routes them to appropriate consistency cluster based on requested model

Strong Consistency Node Cluster

Raft consensus algorithm

Ensures linearizable reads and writes by replicating data with consensus

Eventual Consistency Node Cluster

Gossip protocol

Replicates data asynchronously allowing faster reads with eventual convergence

Persistent Storage

Distributed key-value store (e.g., RocksDB, Cassandra)

Stores data durably on each node

Request Flow

1. Client sends read or write request with consistency preference to API Gateway.

2. API Gateway routes request to Strong Consistency Cluster if strong consistency requested.

3. Strong Consistency Cluster uses Raft to replicate write and confirm commit before responding.

4. Reads in Strong Consistency Cluster query leader node to get latest data.

5. If eventual consistency requested, API Gateway routes request to Eventual Consistency Cluster.

6. Eventual Consistency Cluster applies writes asynchronously and replicates via gossip.

7. Reads in Eventual Consistency Cluster may return stale data but system converges eventually.

8. Both clusters persist data locally to durable storage.

9. Failure detection triggers leader election in Strong Consistency Cluster or repair in Eventual Consistency Cluster.

Database Schema

Entities: - DataItem: key (string), value (blob), version (vector clock or term number) Relationships: - Each DataItem replicated across multiple nodes - Versioning used to detect conflicts in eventual consistency - Strong consistency cluster maintains single leader with committed version - Eventual consistency cluster nodes exchange versions via gossip

Scaling Discussion

Bottlenecks

Strong consistency cluster leader becomes bottleneck under high write load

Network latency impacts consensus protocol performance

Eventual consistency cluster may have stale reads during partitions

Storage capacity limits on each node

API Gateway can become a single point of failure

Solutions

Partition data by key ranges to multiple strong consistency clusters (sharding)

Use faster consensus algorithms or optimize network topology

Implement conflict resolution and version reconciliation in eventual cluster

Add horizontal scaling with auto-scaling storage nodes

Deploy multiple API Gateway instances behind load balancer

Interview Tips

Time: Spend 10 minutes clarifying requirements and constraints, 20 minutes designing architecture and data flow, 10 minutes discussing scaling and trade-offs, 5 minutes summarizing.

Explain difference between strong and eventual consistency with examples

Describe how consensus algorithms ensure strong consistency

Discuss trade-offs between latency, availability, and consistency

Show understanding of replication and failure handling

Highlight how clients can choose consistency per request