Bird
Raised Fist0

Elasticsearch Architecture - Shards, Replicas, Query Routing

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
Elasticsearch Architecture - Shards, Replicas, Query Routing
Design Elasticsearch cluster architecture focusing on shards, replicas, and query routing for scalable, fault-tolerant search
mediumSearch and IndexingFreq: high
Prerequisites:Inverted IndexElasticsearch basicsDistributed systems fundamentals
⚙️
Functional Requirements
  • Index documents distributed across multiple shards for parallelism
  • Maintain replica shards for high availability and fault tolerance
  • Route search queries efficiently to relevant shards and replicas
  • Support near real-time indexing and search with low latency
  • Allow cluster scaling by adding/removing nodes without downtime
  • Provide consistent search results despite shard relocations
📊
Non-Functional Requirements
  • Support indexing throughput of 925K writes/sec (derived from 2B DAU × 40 msgs/user/day)
  • Search query latency P99 under 100ms for typical queries
  • Cluster availability SLA of 99.99% with automatic failover
  • Data retention of 30 days with replication factor of 2
  • Handle peak query load of 2M QPS during traffic spikes
❓
Clarifying Questions to Ask
  • What is the expected document size and average index size per day?
  • What is the replication factor and consistency model for writes?
  • Are queries mostly full-text search, aggregations, or filters?
  • What is the expected cluster size and node hardware specs?
  • Is near real-time search required or eventual consistency acceptable?
  • How frequent are shard rebalancing and cluster topology changes?
Out of Scope
✗ Design of the inverted index data structure itself
✗ Detailed query DSL parsing and scoring algorithms
✗ Security and authentication mechanisms
✗ Cross-cluster replication or multi-region deployment
📌
Assumptions
  • 2 billion daily active users (DAU)
  • Each user generates 40 messages per day
  • Average message size is 100 bytes
  • Replication factor of 2 (1 primary + 1 replica)
⚡
QPS Calculation
write qps2,000,000,000 × 40 / 86,400 ≈ 925,926 writes/sec
peak write qpsAssuming 2× peak traffic: 1,851,852 writes/sec
read qpsAssuming 5× read-to-write ratio: 4,629,630 reads/sec
💾
Storage Estimate
daily raw data2B × 40 × 100 bytes = 8 TB/day
daily indexed dataAssuming 2× indexing overhead: 16 TB/day
30 day retention16 TB × 30 = 480 TB
with replication480 TB × 2 = 960 TB total storage
🖥️
Server Estimate
data nodesAssuming 2 TB usable SSD per node: 960 TB / 2 TB = 480 nodes
query nodesAssuming each node handles 10K QPS: 4,629,630 / 10,000 ≈ 463 nodes
🏗️
Architecture Diagram
Architecture Diagram
          +----------------+
          |  Client/Users  |
          +-------+--------+
                  | Search/Index Requests
                  v
          +-------+--------+
          |  Coordinating  |
          |    Nodes       |
          +-------+--------+
                  | Query Routing
        +---------+---------+
        |                   |
  +-----v-----+       +-----v-----+
  | Data Node |       | Data Node |
  | (Shard 1) |       | (Shard 2) |
  +-----+-----+       +-----+-----+
        |                   |
  +-----v-----+       +-----v-----+
  | Replica   |       | Replica   |
  | Shard 1   |       | Shard 2   |
  +-----------+       +-----------+
🧩
Components
Coordinating Node
Receives client queries and indexing requests, routes them to appropriate shards and replicas
Why thisElasticsearch coordinating nodes handle query fan-out and aggregation, enabling parallel search execution using Elasticsearch's native coordination logic
Why notUsing client-side routing requires clients to maintain cluster topology, increasing complexity and reducing flexibility compared to Elasticsearch's built-in routing
Primary Shard
Stores a subset of indexed documents and handles write operations for that shard
Why thisPrimary shards enable horizontal scaling by partitioning data and distributing indexing load using Elasticsearch's shard mechanism
Why notSingle monolithic index cannot scale beyond single node storage and CPU limits, limiting Elasticsearch cluster scalability
Replica Shard
Maintains a copy of primary shard data for fault tolerance and load balancing read queries
Why thisReplicas provide high availability and improve read throughput by serving search requests in Elasticsearch clusters
Why notNo replication risks data loss and query unavailability during node failures, which Elasticsearch replication mitigates
Cluster State Management
Maintains metadata about shards, nodes, and cluster topology for routing and failover
Why thisElasticsearch uses Zen Discovery and master nodes to maintain consistent cluster state and coordinate shard allocation
Why notExternal coordination services like ZooKeeper add complexity and latency for cluster state updates compared to Elasticsearch's native Zen Discovery
Shard Allocation and Balancing
Distributes shards evenly across data nodes to optimize resource utilization and fault tolerance
Why thisElasticsearch’s built-in shard allocator balances shards based on node load and shard size using its allocation decider framework
Why notManual shard placement is error-prone and does not adapt dynamically to cluster changes, unlike Elasticsearch's automated allocator
🔄
Data Flow
1
Client → Coordinating Node
Send search or index request
Client does not need shard awareness
2
Coordinating Node → Cluster State Manager
Fetch current cluster state and shard locations
Ensures routing decisions use up-to-date topology
3
Coordinating Node → Primary Shards
Route index requests to primary shards
Primary shard handles write and replicates to replicas
4
Coordinating Node → Replica Shards
Route search queries to replicas or primaries based on load
Load balancing improves query throughput and latency
5
Primary Shard → Replica Shards
Replicate indexed data asynchronously
Ensures data durability and availability
6
Replica Shards → Coordinating Node
Return search results
Coordinating node merges partial results
7
Coordinating Node → Client
Return aggregated search response
Client receives unified search results

Practice

(1/5)
1. In the Reddit voting system request flow, at step 4 the Vote Processor reads from the Kafka topic partition keyed by post_id. What state does the Vote Processor maintain after processing a vote event at this step?
easy
A. It updates the in-memory score cache for the post and commits the Kafka offset
B. It sends an immediate notification to all subscribers before updating the score
C. It deletes the vote event from Kafka to prevent reprocessing
D. It marks the vote as delivered but does not update the post's score yet

Solution

  1. Step 1: Understand Vote Processor's role at step 4

    The Vote Processor consumes vote events from Kafka, updates the post's score in an in-memory cache, and commits the offset to mark progress.

  2. Step 2: Eliminate incorrect states

    It marks the vote as delivered but does not update the post's score yet is incorrect because score update happens here. It deletes the vote event from Kafka to prevent reprocessing is invalid as Kafka is immutable; events are not deleted. It sends an immediate notification to all subscribers before updating the score is wrong because notifications happen after score update, not before.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Vote Processor updates score cache and commits Kafka offset [check]
Hint: Vote Processor updates score cache and commits offset [check]
Common Mistakes:
2. A microservices architecture requires low-latency, strongly typed RPC calls between internal services with automatic code generation. Which technology best fits this requirement?
easy
A. SOAP with XML for strict contract enforcement
B. REST over HTTP with JSON payloads for flexibility
C. GraphQL for flexible querying of service data
D. gRPC with Protocol Buffers for efficient binary serialization

Solution

  1. Step 1: Identify serialization and transport needs

    Low-latency and strongly typed RPC require efficient binary serialization and HTTP/2 transport.

  2. Step 2: Evaluate technology fit

    gRPC uses Protocol Buffers for compact binary serialization and HTTP/2 for multiplexed streams, meeting requirements. REST with JSON is flexible but slower and less type-safe. GraphQL is query-focused, not RPC. SOAP is verbose and outdated.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    gRPC + Protobuf -> low-latency, strongly typed RPC [check]
Hint: Binary + HTTP/2 + codegen = gRPC [check]
Common Mistakes:
3. A real-time analytics system must ingest 1 million events per second and provide near-instant query results. Which component best handles the ingestion to balance latency and throughput?
easy
A. A relational database with strong consistency guarantees
B. A batch processing system that aggregates events every minute
C. A distributed log system like Apache Kafka with partitioned topics
D. A single-threaded in-memory queue to ensure ordering

Solution

  1. Step 1: Identify ingestion requirements

    High throughput and low latency ingestion require a scalable, partitioned system that can handle parallel writes.

  2. Step 2: Evaluate components

    Relational DBs (A) can't handle this write volume with low latency; single-threaded queues (C) limit throughput; batch systems (D) increase latency.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Kafka partitions enable parallel ingestion with low latency [check]
Hint: Partitioned logs scale writes and keep latency low [check]
Common Mistakes:
4. For a mobile app requiring real-time notifications at 10x scale, which architecture is better and why?
medium
A. Use WebSockets with a scalable connection broker to handle mobile client limitations
B. Use gRPC streaming for persistent connections to push notifications
C. Use long polling HTTP requests to simulate real-time updates
D. Use periodic polling every 5 seconds to reduce server load

Solution

  1. Step 1: Evaluate mobile client support

    gRPC streaming (Use gRPC streaming for persistent connections to push notifications) is efficient but poorly supported on many mobile platforms, causing compatibility issues.

  2. Step 2: WebSockets with connection broker

    WebSockets (Use WebSockets with a scalable connection broker to handle mobile client limitations) are widely supported on mobile and, with a scalable broker, can handle millions of connections efficiently.

  3. Step 3: Polling drawbacks

    Long polling (Use long polling HTTP requests to simulate real-time updates) and periodic polling (Use periodic polling every 5 seconds to reduce server load) increase latency and server load, unsuitable for real-time at scale.

  4. Final Answer:

    Option A -> Option A

  5. Quick Check:

    WebSockets + broker -> scalable, mobile-friendly real-time [check]
Hint: WebSockets scale better on mobile than gRPC streaming [OK]
Common Mistakes:
5. At 10x scale in user requests and live traffic updates, which component is most likely to fail first and why?
hard
A. The live traffic ingestion pipeline, as message queues saturate and delay update propagation
B. The Road Graph database, due to inability to handle increased write volume for live updates
C. The Routing Engine servers, because shortest path computations become CPU-bound and latency spikes
D. The client-side map rendering, due to increased data volume overwhelming browser memory

Solution

  1. Step 1: Analyze scaling bottlenecks

    Road Graph DB (A) can be sharded; ingestion pipeline (C) can scale horizontally; client rendering (D) is local and less critical.

  2. Step 2: Identify Routing Engine CPU bottleneck

    Routing Engine performs complex shortest path computations per request; 10x load causes CPU saturation and latency spikes, failing first.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Routing Engine CPU-bound under 10x load causes first failure [check]
Hint: Routing Engine CPU saturates first under 10x load [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard