Bird
Raised Fist0

Full-Text Search Architecture - Inverted Index Design from Scratch

HLDmediumInterview freq: common
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Full-Text Search Architecture - Inverted Index Design from Scratch
45 min · senior
senior

You are tasked with designing a full-text search system for a startup with 50 million daily active users. The system must support near real-time indexing and querying of user-generated content, such as posts and comments, with a P99 query latency under 200ms. Data must be encrypted at rest and in transit due to compliance requirements. The system should handle high write throughput from content ingestion and support multi-tenant isolation for different user groups. Offline query capabilities for mobile clients with intermittent connectivity are also required.

Constraints
  • P99 query latency under 200ms at 50M DAU scale
  • Data encrypted at rest and in transit (TLS + disk encryption)
  • Support near real-time indexing with up to 100K writes/sec
  • Multi-tenant isolation for user groups with separate index partitions
  • Offline query support on mobile clients with eventual sync
📋
Guided Steps
1/5
Step 1
Index Data Model and Partitioning
How would you design the inverted index data model and partition it to handle 50M DAU with multi-tenant isolation?
💡 Hint
Consider sharding by tenant and term to balance load and enable efficient queries.
Key Decision
Use Apache Lucene-based inverted index shards partitioned by tenant_id and term hash stored in Elasticsearch clusters to enable horizontal scaling and multi-tenant isolation.
Interviewer Expectation
Mid-level: Names Elasticsearch for inverted index storage. Senior: Explains tenant_id partitioning to isolate data and term hash sharding to balance query load and avoid hot spots.
⚠️Common Wrong Answer
Using a single monolithic index without tenant isolation - leads to noisy neighbor issues and poor query performance under load.
Step 2
Indexing Pipeline and Write Scalability
How will you design the indexing pipeline to support near real-time writes at 100K writes/sec with encryption requirements?
💡 Hint
Think about decoupling ingestion from indexing using a durable, encrypted message queue and batch processing.
Key Decision
Use Apache Kafka with TLS encryption and Kafka Connect to buffer and stream document updates to Elasticsearch bulk APIs, enabling scalable, encrypted, and fault-tolerant ingestion.
Interviewer Expectation
Mid-level: Mentions Kafka for buffering writes. Senior: Details TLS encryption on Kafka, explains bulk indexing to Elasticsearch for throughput, and discusses backpressure handling.
⚠️Common Wrong Answer
Direct synchronous writes to Elasticsearch - causes high latency and poor write scalability under load.
Step 3
Query API Design and Low Latency Retrieval
How would you design the query API to achieve P99 latency under 200ms for full-text search with filters and pagination?
💡 Hint
Consider caching, query routing, and efficient data structures for fast term lookups.
Key Decision
Implement a stateless query service using Elasticsearch's REST API with Redis caching for frequent queries and use filtered queries with pagination to limit result set size and latency.
Interviewer Expectation
Mid-level: Uses Elasticsearch query API. Senior: Explains Redis caching with TTL for hot queries, query routing by tenant shard, and pagination to bound response size.
⚠️Common Wrong Answer
Full index scans or unbounded queries - leads to high latency and query timeouts at scale.
Step 4
Offline Query Support and Synchronization
How will you enable offline query capabilities on mobile clients and synchronize incremental updates efficiently?
💡 Hint
Think about snapshotting index data and using change logs or version vectors for incremental sync.
Key Decision
Generate periodic encrypted index snapshots per tenant stored in AWS S3 with versioning, and use Kafka change logs to stream incremental updates for client delta sync.
Interviewer Expectation
Mid-level: Mentions offline snapshots. Senior: Explains encrypted snapshot storage, versioning for consistency, and Kafka-based incremental sync with conflict resolution.
⚠️Common Wrong Answer
No offline support or full re-downloads - causes excessive bandwidth and poor user experience.
Step 5
Security and Data Encryption
How do you ensure data encryption at rest and in transit throughout the system?
💡 Hint
Consider TLS for network communication and disk encryption for storage layers.
Key Decision
Enable TLS 1.3 for all client-server and inter-service communication, use Elasticsearch encrypted snapshots with AWS KMS-managed keys, and enable disk-level encryption on Kafka brokers and Elasticsearch nodes.
Interviewer Expectation
Mid-level: Mentions TLS and disk encryption. Senior: Details TLS 1.3, KMS key rotation, encrypted snapshots, and encryption key management policies.
⚠️Common Wrong Answer
Relying only on network encryption without disk encryption - risks data exposure if storage is compromised.
🔌
API Design Challenge
Sketch 3 core API endpoints for the full-text search system that cover indexing, querying, and synchronization.
Hints
  • How does the client submit new documents or updates for indexing?
  • How does the client perform search queries with filters and pagination?
  • How does the client synchronize offline query results and handle incremental updates?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Partition inverted index by tenant_id and term hash to enable multi-tenant isolation and load balancing
  • Use Kafka with TLS encryption as a write buffer to decouple ingestion from indexing and support high write throughput
  • Design query API with caching and pagination to meet P99 latency under 200ms
  • Implement offline support with encrypted snapshots and incremental sync using Kafka change logs
  • Enforce encryption at rest and in transit with TLS 1.3 and disk encryption
Good to Have - 30%
  • Use Redis caching with TTL to reduce query latency for frequent queries
  • Use AWS KMS for encryption key management and snapshot encryption
  • Implement backpressure and retry mechanisms in the indexing pipeline
  • Use version vectors for conflict resolution during offline sync
Bonus - 10%
  • Design query routing to tenant-specific shards to reduce query fan-out
  • Implement monitoring and alerting on indexing lag and query latency
  • Use Kafka Connect for seamless integration between ingestion and indexing
Anti-patterns to Avoid
Single monolithic index without tenant isolation - causes noisy neighbor effects and poor scalability under load
Direct synchronous writes to Elasticsearch - leads to high latency and indexing bottlenecks under heavy write load
Unbounded full index scans or unfiltered queries - results in high latency and query timeouts at scale
✅
Model Answer
▼ Reveal

Design a scalable full-text search system for 50M DAU using Elasticsearch with Lucene inverted index shards partitioned by tenant_id and term hash. Use Apache Kafka with TLS encryption and Kafka Connect for durable, encrypted ingestion buffering supporting 100K writes/sec. Implement a stateless query API with Redis caching and pagination to meet P99 latency under 200ms. Enable offline query support via encrypted AWS S3 snapshots with versioning and incremental sync using Kafka change logs and version vectors. Enforce TLS 1.3 for all network communication and disk-level encryption on storage nodes.

Key Decisions
  • Partition inverted index by tenant_id and term hash to isolate tenants and balance query load
  • Use Kafka with TLS encryption as a durable write buffer decoupling ingestion from indexing
  • Bulk index documents into Elasticsearch via Kafka Connect to achieve high write throughput
  • Cache frequent queries in Redis with TTL to reduce query latency and meet P99 SLA
  • Implement pagination and filtered queries to limit result set size and latency
  • Generate periodic encrypted index snapshots per tenant stored in AWS S3 with versioning for offline support
  • Use Kafka change logs and version vectors for efficient incremental synchronization of offline clients
  • Enforce TLS 1.3 for all inter-service and client-server communication and enable disk encryption on Kafka and Elasticsearch nodes
  • Elasticsearch with Lucene inverted index shards partitioned by tenant_id and term hash for scalable multi-tenant search
  • Apache Kafka with TLS encryption as a durable ingestion buffer supporting 100K writes/sec and backpressure handling
  • Redis caching layer for hot query results to meet P99 latency under 200ms
  • AWS S3 encrypted snapshots with versioning for offline index data delivery and incremental sync
  • Stateless query service routing queries to tenant-specific Elasticsearch shards with filtered queries and pagination
  • TLS 1.3 enforced for all network communication and disk-level encryption enabled on Kafka and Elasticsearch nodes
  • Kafka Connect for streaming document updates from ingestion to Elasticsearch bulk APIs
  • Version vectors and Kafka change logs for efficient incremental synchronization of offline clients

Practice

(1/5)
1. A client application needs to fetch deeply nested related data in a single request without multiple round-trips. Which API design approach best supports this requirement?
easy
A. Use GraphQL to request exactly the nested data needed in one query
B. Use WebSockets to stream nested data continuously
C. Use REST with query parameters to fetch all nested data in one endpoint
D. Use REST endpoints with multiple sequential calls to fetch nested resources

Solution

  1. Step 1: Identify nested data fetching needs

    REST typically requires multiple calls or over-fetching to get nested data, causing inefficiency.

  2. Step 2: Recognize GraphQL's single query nested fetch

    GraphQL allows clients to specify nested fields in one request, reducing round-trips and over-fetching.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    GraphQL single query fetches nested data efficiently [check]
Hint: GraphQL fetches nested data in one query [check]
Common Mistakes:
2. For a mobile chat app requiring low latency and offline support, which architecture is best suited to handle message synchronization and delivery?
medium
A. Use WebSockets with server push and no offline caching
B. Use gRPC streaming for real-time message delivery to mobile clients
C. Use REST APIs with client-side local storage and background sync
D. Use polling every few seconds to fetch new messages

Solution

  1. Step 1: Analyze mobile client constraints

    Mobile clients often have intermittent connectivity and limited support for persistent streaming (gRPC streaming in Use gRPC streaming for real-time message delivery to mobile clients is unreliable on mobile). WebSockets without offline caching (Use WebSockets with server push and no offline caching) fail offline support. Polling (Use polling every few seconds to fetch new messages) adds latency and battery drain.

  2. Step 2: Evaluate REST with local storage

    REST APIs combined with client-side local storage and background sync enable offline message access and eventual consistency with low latency.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    REST + local storage -> offline support and sync [check]
Hint: Offline support needs local storage, not persistent streaming [check]
Common Mistakes:
3. If the Kafka topic used for driver location updates is deleted accidentally, what is the immediate failure observed in the system?
medium
A. Driver location updates are lost permanently with no retry, causing stale driver positions
B. The system automatically switches to a backup topic with no data loss
C. Driver apps crash due to inability to send location updates
D. Matching service blocks waiting for location updates, causing request timeouts

Solution

  1. Step 1: Understand Kafka topic role

    Kafka topic buffers and streams driver location updates to downstream services.

  2. Step 2: Trace failure impact

    Deleting the topic causes producer writes to fail silently or drop messages; no automatic backup switch; driver apps continue sending but updates are lost; matching service receives stale data but does not block.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Kafka topic deletion causes permanent loss of location updates [check]
Hint: No Kafka topic means lost updates, no automatic failover [check]
Common Mistakes:
4. At 10x scale, which component of the key-value store is most likely to become the first bottleneck and why?
hard
A. Vector clocks, because version metadata grows exponentially with concurrent writes
B. Gossip protocol, because membership message overhead grows quadratically with node count
C. Consistent hashing ring, because rebalancing keys requires global coordination
D. Storage nodes, because disk I/O cannot keep up with increased write throughput

Solution

  1. Step 1: Analyze storage node scaling

    At 10x write throughput, disk I/O bandwidth and latency become the bottleneck before metadata or gossip overhead.

  2. Step 2: Compare other components

    Gossip overhead grows but can be optimized; vector clocks metadata grows slowly; consistent hashing ring updates are incremental.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Disk I/O limits storage node write throughput at large scale [check]
Hint: Disk I/O limits storage node throughput at scale [OK]
Common Mistakes:
5. At 10x scale, which component in the faceted search system is most likely to fail first and why?
hard
A. The Query Coordinator, because it cannot merge partial results fast enough under high concurrency
B. The indexing pipeline, because it cannot keep up with the volume of document updates
C. The shard-level cache, because memory pressure causes frequent evictions and cache misses spike
D. The shard nodes, because inverted index lookups become CPU-bound and cannot handle query volume

Solution

  1. Step 1: Analyze bottlenecks at 10x query scale

    Shard nodes perform CPU-intensive inverted index lookups; query volume increase causes CPU saturation first.

  2. Step 2: Compare other components

    Query Coordinator merges are parallelizable; cache evictions increase but cache can be scaled; indexing pipeline is async and can be scaled horizontally.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Shard nodes CPU-bound -> first failure point at scale [check]
Hint: Shard nodes CPU-bound first at 10x scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard