Bird
Raised Fist0

Elasticsearch Architecture - Shards, Replicas, Query Routing

HLDmediumInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Elasticsearch Architecture - Shards, Replicas, Query Routing
45 min · senior
senior

You are designing the Elasticsearch cluster architecture for a startup with 50 million daily active users. The system must support full-text search and analytics on petabytes of data with P99 query latency under 200ms. Data must be encrypted at rest and in transit. The system should tolerate node failures without data loss or downtime. Offline query delivery is required for mobile clients with intermittent connectivity.

Constraints
  • P99 query latency < 200ms at 50M DAU scale
  • Data encrypted at rest and in transit (TLS + disk encryption)
  • Support offline query delivery with eventual consistency
  • Cluster must tolerate node failures with zero data loss
  • Index size expected to grow to petabytes over 12 months
📋
Guided Steps
1/6
Step 1
Shard Strategy and Data Partitioning
How would you design the shard layout to balance query latency, indexing throughput, and fault tolerance?
💡 Hint
Consider Elasticsearch’s primary shards and replica shards, and how shard count affects parallelism and recovery.
Key Decision
Use Elasticsearch primary shards to horizontally partition data by document ID hash, with 1-2 replicas per primary shard to ensure fault tolerance and read scalability.
Interviewer Expectation
Mid: Names primary and replica shards. Senior: Explains shard count trade-offs on query latency, indexing throughput, and recovery time, and why too many shards increase cluster overhead.
⚠️Common Wrong Answer
Using a very high shard count (e.g., thousands) to increase parallelism - leads to excessive cluster state size and slower recovery.
Step 2
Replica Placement and Fault Tolerance
How do you ensure replicas provide high availability and data durability while minimizing cross-node network overhead?
💡 Hint
Think about Elasticsearch’s shard allocation awareness and how replicas are distributed across nodes and racks.
Key Decision
Configure Elasticsearch shard allocation awareness to place replicas on different physical racks or availability zones, ensuring no single point of failure and minimizing correlated failures.
Interviewer Expectation
Mid: Mentions replicas on different nodes. Senior: Explains rack-awareness and zone-awareness settings to avoid correlated failures and reduce network latency during replication.
⚠️Common Wrong Answer
Placing replicas randomly without awareness - risks data loss if multiple replicas reside on the same failure domain.
Step 3
Query Routing and Aggregation
How does the query coordinator route search requests to shards and aggregate partial results efficiently within the 200ms P99 latency budget?
💡 Hint
Consider Elasticsearch’s coordinating node role and how it merges shard-level results.
Key Decision
Use Elasticsearch coordinating nodes to route queries to all relevant primary and replica shards in parallel, merging results client-side with early termination and result windowing to meet latency SLAs.
Interviewer Expectation
Mid: Describes coordinating node role. Senior: Explains parallel shard fan-out, replica preference for load balancing, and techniques like early termination and result windowing to reduce latency.
⚠️Common Wrong Answer
Querying only primary shards to reduce complexity - reduces fault tolerance and increases query latency under load.
Step 4
Replica Synchronization and Consistency
How do replicas stay consistent with primary shards, and how is write durability guaranteed under node failures?
💡 Hint
Think about Elasticsearch’s translog and replication protocol.
Key Decision
Elasticsearch uses a write-ahead transaction log (translog) on primary shards; writes are replicated asynchronously to replicas with a quorum-based write consistency setting to guarantee durability before acknowledging clients.
Interviewer Expectation
Mid: Mentions translog and replication. Senior: Explains translog flush frequency, replica recovery process, and how write consistency levels affect durability and latency trade-offs.
⚠️Common Wrong Answer
Acknowledging writes immediately after primary commit without waiting for replicas - risks data loss if primary fails before replication.
Step 5
Encryption and Security
How do you ensure data is encrypted at rest and in transit without impacting query performance significantly?
💡 Hint
Consider Elasticsearch’s native encryption features and TLS configurations.
Key Decision
Enable Elasticsearch’s native TLS encryption for all inter-node and client communications, and use disk-level encryption (e.g., dm-crypt or cloud provider encryption) for data at rest to meet compliance without modifying query paths.
Interviewer Expectation
Mid: Mentions TLS and disk encryption. Senior: Explains TLS mutual authentication, key rotation strategies, and how encryption overhead is minimized by hardware acceleration and avoiding application-level encryption.
⚠️Common Wrong Answer
Encrypting data at the application layer before indexing - causes large query latency overhead and disables Elasticsearch’s internal optimizations.
Step 6
Offline Query Delivery for Mobile Clients
How would you design offline query delivery to mobile clients with intermittent connectivity while maintaining eventual consistency?
💡 Hint
Think about snapshotting, incremental updates, and client-side caching.
Key Decision
Use Elasticsearch snapshot and restore APIs to generate periodic index snapshots stored in object storage; mobile clients download snapshots and receive incremental delta updates via a lightweight sync service using Kafka for change events.
Interviewer Expectation
Mid: Suggests snapshotting. Senior: Details incremental sync via Kafka change streams, client-side merge conflict resolution, and trade-offs between snapshot frequency and data freshness.
⚠️Common Wrong Answer
Relying solely on real-time queries from mobile clients - fails with intermittent connectivity and increases backend load.
🔌
API Design Challenge
Sketch 3 core Elasticsearch API endpoints to support indexing, querying, and replica synchronization.
Hints
  • How does the client submit documents for indexing and ensure durability?
  • How does the query coordinator route queries to shards and aggregate results?
  • How do replicas synchronize with primary shards to maintain consistency?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Shard partitioning strategy balancing query latency and indexing throughput
  • Replica placement with failure domain awareness
  • Query routing via coordinating nodes with parallel shard fan-out
  • Replica synchronization using translog and quorum write consistency
  • Encryption at rest and in transit
Good to Have - 30%
    Bonus - 10%
      Anti-patterns to Avoid
      ✅
      Model Answer
      ▼ Reveal

      This design leverages Elasticsearch's primary and replica shards to partition and replicate data for scalability and fault tolerance. Coordinating nodes route queries in parallel to shards, merging results efficiently to meet latency SLAs. Durability is ensured via the translog and quorum-based replication. Security is enforced with TLS for in-transit encryption and disk-level encryption for data at rest. Offline query delivery is supported through snapshots and incremental Kafka-based syncs.

      Key Decisions
      • Use Elasticsearch primary shards to horizontally partition data by document ID hash
      • Configure 1-2 replicas per primary shard for fault tolerance and read scalability
      • Enable shard allocation awareness to place replicas across different racks or availability zones
      • Use coordinating nodes to fan out queries to all relevant shards and merge results with early termination and result windowing
      • Implement quorum-based write consistency with translog to guarantee durability before client acknowledgment
      • Enable Elasticsearch native TLS encryption for all inter-node and client communications
      • Use disk-level encryption (e.g., dm-crypt or cloud provider encryption) for data at rest
      • Leverage Elasticsearch snapshot and restore APIs combined with Kafka change streams for offline query delivery and incremental sync
      • Elasticsearch primary shards partition data by document ID hash for horizontal scaling and parallelism
      • Replica shards placed with shard allocation awareness across racks/zones to ensure fault tolerance
      • Coordinating nodes fan out queries to all relevant shards and merge results with early termination to meet 200ms P99 latency
      • Write durability ensured by Elasticsearch translog and quorum-based replication before client acknowledgment
      • TLS encryption enabled for all inter-node and client communications; disk-level encryption for data at rest
      • Offline query delivery via Elasticsearch snapshots stored in object storage plus incremental Kafka-based change event sync

      Practice

      (1/5)
      1. In the task scheduler's execution flow, at Step 4 the Worker Node polls the distributed priority queue for the next task. What happens if the task's scheduled time has not yet arrived?
      easy
      A. The Worker Node marks the task as failed due to premature execution
      B. The Worker Node re-queues the task with the same priority and waits
      C. The Worker Node skips the task and polls for the next available task
      D. The Worker Node immediately executes the task ignoring the schedule

      Solution

      1. Step 1: Understand Worker Node behavior at Step 4

        The Worker Node must respect the scheduled execution time; executing early (A) violates correctness; re-queuing (B) risks duplicate tasks; skipping (C) ignores the task indefinitely.

      2. Step 2: Identify failure handling

        Marking the task as failed (D) triggers retry or alert mechanisms, ensuring no premature execution and visibility into scheduling issues.

      3. Final Answer:

        Option A -> Option A

      4. Quick Check:

        Premature tasks are failed to prevent early execution [check]
      Hint: Fail early if task scheduled time not reached [check]
      Common Mistakes:
      2. In a system using TCP for file transfer, at step 4 the TCP sender retransmits a packet after a timeout. What is the state of the sender's congestion window at this step?
      easy
      A. The congestion window is increased to allow faster sending
      B. The congestion window is doubled to compensate for lost packets
      C. The congestion window remains unchanged until ACKs arrive
      D. The congestion window is reduced to the initial slow-start threshold

      Solution

      1. Step 1: Understand TCP congestion control on timeout

        When a retransmission timeout occurs, TCP interprets it as network congestion and reduces the congestion window to the slow-start threshold.

      2. Step 2: Confirm congestion window behavior

        The congestion window is reset to a lower value to reduce sending rate, avoiding further congestion.

      3. Final Answer:

        Option D -> Option D

      4. Quick Check:

        Timeout triggers congestion window reduction to slow-start threshold [check]
      Hint: Timeout -> congestion window reset to slow-start threshold [check]
      Common Mistakes:
      3. In a system generating KSUIDs, at step 4 the ID generator component appends a random payload after the timestamp. What is the state of the generated ID at this step?
      easy
      A. The ID is a UUIDv4 embedded inside the KSUID format
      B. The ID is fully deterministic and only contains the timestamp
      C. The ID contains a timestamp prefix followed by a cryptographically secure random payload
      D. The ID is a sequential number incremented per node

      Solution

      1. Step 1: Understand KSUID structure at step 4

        KSUIDs combine a 4-byte timestamp prefix with a 16-byte random payload to ensure uniqueness and sortability.

      2. Step 2: Eliminate incorrect states

        Deterministic timestamp only (A) lacks uniqueness; UUIDv4 (C) is unrelated; sequential number (D) is not part of KSUID design.

      3. Final Answer:

        Option C -> Option C

      4. Quick Check:

        KSUID = timestamp + random payload for uniqueness and sortability [check]
      Hint: KSUID = timestamp prefix + random payload [check]
      Common Mistakes:
      4. Without the Kafka message queue, what exactly breaks in the chat delivery flow?
      medium
      A. Alice's send blocks until Bob responds; if Bob's server crashes, message is lost with no retry
      B. Messages are encrypted incorrectly causing security breaches
      C. Messages are duplicated multiple times due to lack of deduplication
      D. The UI shows messages as delivered even if Bob never received them

      Solution

      1. Step 1: Trace message flow without Kafka

        Without Kafka, Chat Server A must synchronously contact Bob's server. If Bob's server is slow, Alice's send blocks; if Bob's server crashes, no retry or persistence exists.

      2. Step 2: Eliminate unrelated failure modes

        Encryption issues (A) unrelated to Kafka. Duplication (C) and UI false delivery (D) are mitigated by Kafka's ordering and delivery semantics but not the primary failure here.

      3. Final Answer:

        Option A -> Option A

      4. Quick Check:

        No Kafka -> synchronous send blocks and message loss on Bob crash [check]
      Hint: No Kafka -> synchronous send blocks and message lost if recipient crashes [check]
      Common Mistakes:
      5. At 10x scale, which component in Netflix's adaptive streaming pipeline is most likely to fail first and why?
      hard
      A. Personalization service because recommendation model retraining becomes too slow
      B. Origin servers because storage I/O bandwidth saturates under high request volume
      C. CDN edge servers because WebSocket connections for adaptive bitrate are stateful and cannot be easily sharded
      D. Client devices because decoding multiple bitrate streams consumes excessive CPU

      Solution

      1. Step 1: Identify stateful bottlenecks at scale

        Personalization retraining (Personalization service because recommendation model retraining becomes too slow) can be scaled horizontally with distributed training. Origin storage (Origin servers because storage I/O bandwidth saturates under high request volume) can be scaled with sharding and CDN caching. Client devices (Client devices because decoding multiple bitrate streams consumes excessive CPU) are outside server control.

      2. Step 2: Analyze CDN edge server scaling challenges

        WebSocket connections for adaptive streaming are stateful and bound to specific edge servers, limiting horizontal scaling and resharding, causing edge server overload at 10x scale.

      3. Final Answer:

        Option C -> Option C

      4. Quick Check:

        Stateful WebSocket limits CDN edge server scalability [check]
      Hint: Stateful WebSocket limits CDN edge scaling first [check]
      Common Mistakes:
      Recommended

      Ready to prove it?

      Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

      10 questions~5 mineasy hard