Bird
Raised Fist0

Design Google Search - Crawling, Indexing, PageRank, Query Processing

HLDhardInterview freq: high
Choose your preparation mode4 modes available
🏗️
Design Google Search - Crawling, Indexing, PageRank, Query Processing
Design a scalable, low-latency web search engine handling billions of pages and queries
hardSearch and DiscoveryFreq: high
Prerequisites:Inverted IndexElasticsearch
⚙️
Functional Requirements
  • Continuously crawl billions of web pages globally with freshness guarantees
  • Parse and extract content, metadata, and links from crawled pages
  • Build and maintain a distributed inverted index supporting multi-language text
  • Compute PageRank scores periodically to rank pages by importance
  • Process user queries with low latency (sub-100ms P99) returning ranked results
  • Support incremental indexing and near real-time updates for fresh content
📊
Non-Functional Requirements
  • Support 2 billion daily active users generating ~1 billion queries per day
  • Handle peak query QPS of ~925,000 with 100ms P99 latency
  • Ensure 99.99% availability for query serving
  • Index storage must scale to hundreds of petabytes with replication
  • Crawling system must sustain continuous high throughput with fault tolerance
❓
Clarifying Questions to Ask
  • What is the expected freshness SLA for crawled pages (e.g., daily, hourly)?
  • What languages and content types must the index support?
  • Are we supporting real-time indexing or batch-only updates?
  • What is the expected query complexity (simple keyword vs. complex boolean)?
  • Is PageRank computed offline or updated incrementally?
  • What is the expected data center and geographic distribution?
Out of Scope
✗ Personalized search ranking and user profiling
✗ Ad serving and monetization
✗ Mobile app UI design
✗ Voice search and natural language understanding
📌
Assumptions
  • 2 billion daily active users (DAU)
  • Average 40 queries per user per day
  • Average query size 100 bytes
  • Average web page size 100 KB
  • Crawl 20 billion pages with average update frequency of 30 days
⚡
QPS Calculation
write qps20B pages / 30 days / 86400 sec ≈ 7,716 writes/sec for crawling
peak write qpsAssuming bursts, peak crawl writes ≈ 20,000 writes/sec
read qps2B users × 40 queries/day / 86400 sec ≈ 925,926 queries/sec
💾
Storage Estimate
daily storage20B pages × 100 KB / 30 days ≈ 66.7 TB/day
retentionStore 30 days of crawled data → 66.7 TB × 30 = 2 PB
index storageInverted index ~30% of raw data → 0.3 × 2 PB = 600 TB
with replication3x replication → 600 TB × 3 = 1.8 PB for index
🖥️
Server Estimate
crawler nodesAssuming 1 crawler node handles 100 pages/sec → 20B/30/86400/100 ≈ 77,160 nodes
indexing nodesElasticsearch cluster with 1000 nodes to handle indexing and query load with replication
🏗️
Architecture Diagram
Architecture Diagram
User Queries
     |
     v
Query Router (Load Balancer)
     |
     v
Distributed Query Processing Cluster (Elasticsearch)
     |
     v
Inverted Index Storage (HDFS + Elasticsearch Shards)
     ^
     |
Indexing Pipeline
     ^
     |
Crawling System (Distributed Crawlers + Kafka)
     |
     v
Web Pages (Internet)
     |
     v
PageRank Computation Cluster (Batch on Hadoop/Spark)
     |
     v
PageRank Scores stored in Index
🧩
Components
Distributed Crawling System
Continuously fetch billions of web pages with politeness and freshness
Why thisApache Nutch with Kafka for scalable, fault-tolerant crawl queue management and distributed fetchers
Why notCustom single-threaded crawler lacks scalability and fault tolerance at 20K writes/sec
Kafka
Decouple crawling fetch events from parsing and indexing pipelines asynchronously
Why thisKafka supports high throughput (millions msgs/sec), partitioning, and durability for crawl data streams
Why notRabbitMQ lacks partitioning and throughput needed for 20K+ writes/sec crawl events
Parsing and Indexing Pipeline
Parse HTML, extract text and links, build inverted index entries
Why thisApache Hadoop + Spark for scalable batch processing of crawl data and Elasticsearch for indexing
Why notRelational DBs like PostgreSQL cannot handle petabyte-scale inverted index writes
Elasticsearch Cluster
Store and serve inverted index shards for low-latency query processing
Why thisElasticsearch supports distributed inverted index, full-text search, and near real-time updates
Why notSolr lacks native near real-time indexing and elastic scaling at 1M QPS
PageRank Computation Cluster
Compute PageRank scores periodically using link graph data
Why thisApache Spark on Hadoop for iterative graph processing at petabyte scale
Why notGraph databases like Neo4j do not scale to billions of nodes and edges efficiently
Query Router (Load Balancer)
Distribute incoming queries evenly across Elasticsearch nodes
Why thisNGINX or Envoy provides high throughput, low latency load balancing with health checks
Why notDNS round-robin lacks health checks and fine-grained load balancing
🔄
Data Flow
1
Distributed Crawlers → Kafka
Push fetched raw web pages and metadata
Kafka decouples crawl fetch from downstream processing
2
Kafka → Parsing and Indexing Pipeline
Consume crawl data, parse HTML, extract text and links
Batch jobs transform raw pages into indexable documents
3
Parsing and Indexing Pipeline → Elasticsearch Cluster
Index parsed documents into distributed inverted index shards
Supports near real-time updates for fresh content
4
PageRank Computation Cluster → Elasticsearch Cluster
Update PageRank scores in index documents
PageRank scores influence ranking during query processing
5
User Queries → Query Router
Route queries to Elasticsearch nodes
Load balancing ensures even query distribution
6
Query Router → Elasticsearch Cluster
Execute full-text search and ranking
Return top ranked results with PageRank and relevance scores
7
Elasticsearch Cluster → User
Send ranked search results
Ensure sub-100ms P99 latency for query responses
🗄️
WebPages
PK: domain
Stores metadata and content hashes for crawled web pages
🗄️
InvertedIndex
PK: term
Stores term postings lists for full-text search
🔌
SearchQuery
Request
{
  "query": "string",
  "language": "string",
  "max_results": "int"
}
Response
{
  "results": "List<SearchResult>",
  "total_hits": "int"
}
Handles user search queries with ranking and pagination
🔌
PageRankUpdate
Request
{
  "doc_id": "string",
  "pagerank_score": "float"
}
Response
{
  "status": "string"
}
Updates PageRank scores in the index
🔌
CrawlStatus
Request
{
  "domain": "string"
}
Response
{
  "last_crawled": "timestamp",
  "status": "string"
}
Provides crawl freshness and status for a domain

Practice

(1/5)
1. A distributed key-value store must remain operational during network partitions but can tolerate stale reads temporarily. Which system design choice best supports this requirement?
easy
A. Use a CP system that blocks writes during partitions to maintain consistency
B. Use a CA system that rejects requests during partitions to maintain availability and consistency
C. Use a synchronous replication system that waits for all replicas before responding
D. Use an AP system that serves reads and writes with eventual consistency during partitions

Solution

  1. Step 1: Identify CAP constraints

    During network partitions, a system must choose between consistency and availability.
  2. Step 2: Match system behavior

    AP systems prioritize availability and partition tolerance, allowing stale reads but remaining operational.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    AP systems allow availability during partitions with eventual consistency [check]
Hint: AP favors availability over consistency during partitions [check]
Common Mistakes:
2. In a distributed file system request flow, at Step 4 the Metadata Server responds to the client with block locations. What state does the client hold immediately after Step 4?
easy
A. Client has cached file data blocks locally
B. Client holds metadata including block locations but has not contacted Data Nodes yet
C. Client has updated the Metadata Server with new file writes
D. Client has established persistent connections to all Data Nodes

Solution

  1. Step 1: Trace Step 4 response

    At Step 4, Metadata Server returns block locations (metadata) to the client; client does not yet have file data.
  2. Step 2: Confirm client state

    Client holds metadata (block locations) but must contact Data Nodes separately to read/write data blocks.
  3. Final Answer:

    Option B -> Option B
  4. Quick Check:

    Client holds metadata, not data, immediately after Step 4 [check]
Hint: Client gets block locations before contacting Data Nodes [check]
Common Mistakes:
3. In a system generating IDs using a custom ID service, at step 3 the ID generator assigns a timestamp-based prefix before appending a sequence number. What happens if the sequence number reaches its max value before the timestamp changes?
easy
A. The ID generator switches to UUID generation to avoid sequence overflow
B. The ID generator blocks until the timestamp increments to avoid collisions
C. The ID generator resets the sequence number to zero immediately, risking duplicate IDs
D. The ID generator skips the timestamp and uses only the sequence number

Solution

  1. Step 1: Trace ID generation at step 3

    The timestamp prefix combined with a sequence number ensures uniqueness within the same timestamp.
  2. Step 2: Understand sequence number overflow handling

    When the sequence number maxes out, the generator must wait for the timestamp to increment to avoid duplicates, thus blocking new ID generation temporarily.
  3. Final Answer:

    Option B -> Option B
  4. Quick Check:

    Sequence overflow -> block until timestamp changes [check]
Hint: Sequence maxed -> wait for timestamp tick [check]
Common Mistakes:
4. For timeline generation at scale, which architecture better handles users with millions of followers and why?
medium
A. Pure pull-based: generate timelines on-demand by querying follow graph and tweets
B. Push-based fan-out: write each tweet to all followers' timelines synchronously
C. Hybrid approach: push fan-out for users with few followers, pull on read for celebrities
D. Store timelines in a global sorted set updated asynchronously for all users

Solution

  1. Step 1: Analyze push vs pull trade-offs

    Push fan-out is efficient for low-follower users but expensive for celebrities; pull is costly on read latency.
  2. Step 2: Evaluate hybrid approach

    Hybrid uses push for most users, pull for celebrities, balancing latency and write amplification.
  3. Step 3: Eliminate other options

    Pure push (A) causes huge write amplification; pure pull (C) causes high read latency; global sorted set (D) is hard to scale and update asynchronously.
  4. Final Answer:

    Option C -> Option C
  5. Quick Check:

    Hybrid push-pull balances write cost and read latency [check]
Hint: Hybrid push-pull fits follower count extremes [check]
Common Mistakes:
5. At 10x scale, which component is most likely to become the bottleneck first in the LLM-powered support bot, and why?
hard
A. The Intent Classifier service, due to synchronous LLM inference latency under high concurrency
B. The human agent pool, because fallback requests increase linearly with user queries
C. The message queue, because it cannot handle the increased throughput of fallback messages
D. The database storing user session states, due to write contention at scale

Solution

  1. Step 1: Identify scaling challenges for each component

    Human agents (A) scale linearly but can be increased with hiring. Message queue (C) scales well horizontally. Database (D) can be sharded.
  2. Step 2: Analyze Intent Classifier bottleneck

    Synchronous LLM inference is compute-intensive and latency-sensitive, causing concurrency limits and queuing delays at 10x load.
  3. Step 3: Confirm first failure point

    Intent Classifier service hits CPU/GPU resource limits first, blocking downstream components.
  4. Final Answer:

    Option A -> Option A
  5. Quick Check:

    Synchronous LLM inference latency limits concurrency first [check]
Hint: LLM inference latency blocks concurrency at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard