Bird
Raised Fist0

Design Google Search - Crawling, Indexing, PageRank, Query Processing

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/7
Distributed Crawling System

Fetch new web page

Crawls a newly discovered URL and fetches HTML content

Protocol:HTTP
BeforeURL discovered but page not fetched
AfterRaw HTML content fetched and ready for processing
Kafka

Publish crawled page

Pushes raw HTML content message to Kafka topic for indexing

Protocol:Kafka Publish
BeforeRaw HTML content only in crawler memory
AfterPage content durably stored in Kafka topic
Parsing and Indexing Pipeline

Consume and parse page

Reads page from Kafka, extracts text and metadata, builds index entries

Protocol:Kafka Consume
BeforePage content in Kafka topic, not yet parsed
AfterParsed page data ready to update index
Elasticsearch Cluster

Update inverted index

Receives parsed data and updates inverted index for search

Protocol:REST API
BeforeIndex missing new page data
AfterIndex updated with new page terms and metadata
PageRank Computation Cluster

Compute PageRank

Processes link graph updates and recalculates PageRank scores

Protocol:RPC
BeforePageRank scores outdated for new page
AfterPageRank scores updated including new page
Query Router (Load Balancer)

Receive user query

Accepts user search query and routes to Elasticsearch

Protocol:REST API
BeforeUser query not yet processed
AfterQuery forwarded to Elasticsearch cluster
Elasticsearch Cluster

Serve search results

Executes query using inverted index and PageRank scores, returns results

Protocol:REST API
BeforeQuery received but results not computed
AfterSearch results returned to user via Query Router
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────────────────────┐
│ Distributed Crawling System   │
│  - Fetches web pages          │
└───────────────┬───────────────┘
                │ Kafka Publish (Kafka)
                ▼
┌───────────────────────────────┐
│ Kafka                         │
│  - Message queue for pipeline │
└───────────────┬───────────────┘
                │ Kafka Consume (Kafka)
                ▼
┌───────────────────────────────┐
│ Parsing and Indexing Pipeline  │
│  - Parses pages, builds index  │
└───────────────┬───────────────┘
                │ Index Update (REST)
                ▼
┌───────────────────────────────┐
│ Elasticsearch Cluster          │
│  - Stores inverted index       │
└───────────────┬───────────────┘
                │ PageRank Data (RPC)
                ▼
┌───────────────────────────────┐
│ PageRank Computation Cluster   │
│  - Computes page ranks          │
└───────────────┬───────────────┘
                │ PageRank Update (RPC)
                ▼
┌───────────────────────────────┐
│ Query Router (Load Balancer)   │
│  - Routes user queries          │
└───────────────┬───────────────┘
                │ Query (REST)
                ▼
┌───────────────────────────────┐
│ Elasticsearch Cluster          │
│  - Returns search results       │
└───────────────────────────────┘
Design Rationale

This architecture illustrates the vertical flow of a web page through Google Search's core systems. The Distributed Crawling System fetches raw HTML, which is then published to Kafka, decoupling crawling from indexing. The Parsing and Indexing Pipeline consumes from Kafka, extracts content, and updates the Elasticsearch Cluster's inverted index. Meanwhile, the PageRank Computation Cluster recalculates page importance scores based on link graphs. User queries arrive at the Query Router, which forwards them to Elasticsearch to retrieve ranked results. This design ensures scalability by separating concerns with Kafka, enables near real-time indexing, and integrates PageRank scores to improve result relevance.

Primary Focus

✓

Practice

(1/5)
1. In the digital wallet system's transaction processing flow, at Step 4 the Ledger Service applies the debit and credit entries. What happens if the Ledger Service detects a mismatch in the double-entry records at this step?
easy
A. The Ledger Service retries the transaction indefinitely until the entries match
B. The Ledger Service marks the transaction as failed and triggers an alert for manual reconciliation
C. The Ledger Service ignores the mismatch and commits the debit entry only
D. The Ledger Service rolls back the entire transaction and deletes all related entries

Solution

  1. Step 1: Understand Ledger Service's double-entry validation

    The Ledger Service enforces double-entry consistency; mismatches indicate data corruption or bugs.

  2. Step 2: Evaluate failure handling

    Retrying indefinitely (A) risks blocking; ignoring mismatch (C) breaks accounting; deleting entries (D) risks data loss. Marking failed and alerting (B) enables safe manual reconciliation.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Mismatch triggers failure mark and alert -> safe reconciliation [check]
Hint: Mismatch triggers failure mark and alert, not silent retry [OK]
Common Mistakes:
2. In the file sync system's upload flow, at Step 4 the Delta Upload Worker receives chunk hashes from the client. What does the worker do when it finds a chunk hash already stored on the server?
easy
A. Skips uploading that chunk and marks it as already present
B. Queues the chunk for re-chunking to optimize storage
C. Deletes the existing chunk and replaces it with the new one
D. Requests the client to resend the chunk data to verify integrity

Solution

  1. Step 1: Trace Delta Upload Worker behavior at Step 4

    When the worker finds a chunk hash already stored, it skips re-uploading the chunk to save bandwidth and marks it as present.

  2. Step 2: Eliminate incorrect behaviors

    Requesting resend wastes bandwidth; deleting existing chunks breaks deduplication; re-chunking is unrelated to upload step.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Delta Upload Worker skips existing chunks to avoid redundant upload [check]
Hint: Skip chunks with known hashes to save bandwidth [check]
Common Mistakes:
3. For a high-read, low-write system, which caching strategy best balances freshness and performance?
medium
A. Use write-through cache to update cache synchronously on every write
B. Disable caching and rely on database read replicas for freshness
C. Use write-back cache to batch writes and update cache asynchronously
D. Use cache-aside pattern where application loads cache on demand and invalidates on writes

Solution

  1. Step 1: Analyze write frequency and freshness needs

    High-read, low-write favors lazy loading and invalidation to avoid cache update overhead.

  2. Step 2: Evaluate caching strategies

    Write-through adds write latency; write-back risks stale data; cache-aside balances freshness and performance.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Cache-aside suits high-read, low-write with freshness [check]
Hint: Cache-aside balances freshness and performance [check]
Common Mistakes:
4. At 10x scale, which component in an idempotent payment API system is most likely to become the bottleneck first, and why?
hard
A. The payment processor, because it cannot handle increased transaction volume
B. The client SDK, as it struggles to generate unique idempotency keys at scale
C. The API gateway, because it cannot route 10x more requests efficiently
D. The idempotency key store, due to synchronous writes and read-after-write consistency requirements

Solution

  1. Step 1: Identify idempotency store workload at scale

    Idempotency store must synchronously write and read keys with strong consistency, creating a high throughput bottleneck.

  2. Step 2: Compare other components

    Payment processors can be scaled horizontally; API gateways are stateless and scalable; client SDKs generating keys is lightweight.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Synchronous idempotency store writes limit scaling first [check]
Hint: Idempotency store sync writes throttle scaling first [check]
Common Mistakes:
5. At 10x scale, which component in the Redlock system is most likely to fail first and why?
hard
A. Redis masters due to increased memory usage from many locks
B. Lock expiration timers due to clock drift causing inconsistent lock validity
C. Network links between clients and Redis masters due to increased lock request volume
D. Clients due to increased network latency causing lock acquisition timeouts

Solution

  1. Step 1: Identify bottleneck at 10x scale

    Increased lock requests increase network latency and contention on clients acquiring locks.

  2. Step 2: Analyze impact on lock acquisition

    Higher latency causes clients to miss lock acquisition timeouts, reducing throughput and increasing retries.

  3. Step 3: Compare other components

    Redis masters scale better with sharding; network links and timers less likely first failure.

  4. Final Answer:

    Option D -> Option D

  5. Quick Check:

    Clients hit network latency -> lock timeouts -> first failure at scale [check]
Hint: Clients face latency -> lock timeouts -> first failure at scale [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard