Bird
Raised Fist0

Design a Web Crawler - Scheduler, Politeness Policy, Duplicate Detection

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/7
URL Frontier

Emit next URL to crawl

URL Frontier publishes next URL to Scheduler Node via Kafka

Protocol:Kafka
BeforeURL Frontier contains multiple URLs queued by priority
AfterNext URL message sent to Scheduler Node, URL marked as in-progress
Scheduler Node

Request politeness check

Scheduler Node calls Politeness Enforcer to verify crawl delay for domain

Protocol:RPC
BeforeScheduler Node has URL ready but must respect crawl delays
AfterScheduler Node receives permission or delay from Politeness Enforcer
Politeness Enforcer

Check domain crawl delay

Politeness Enforcer verifies last crawl time for domain and returns allowed status

Protocol:REST API
BeforePoliteness Enforcer has last crawl timestamps per domain
AfterPoliteness Enforcer returns allowed or delayed status to Scheduler Node
Scheduler Node

Query duplicate detection

Scheduler Node queries Duplicate Detection DB to check if URL was crawled before

Protocol:DB Query
BeforeScheduler Node has URL approved by politeness policy
AfterScheduler Node receives duplicate check result
Duplicate Detection DB

Lookup URL fingerprint

Duplicate Detection DB checks if URL fingerprint exists to avoid re-crawling

Protocol:DB Query
BeforeDuplicate Detection DB stores fingerprints of all crawled URLs
AfterDuplicate Detection DB returns found or not found status
Scheduler Node

Dispatch URL to fetcher

Scheduler Node sends URL to Fetcher Pool for actual page retrieval

Protocol:RPC
BeforeScheduler Node confirmed URL is allowed and not duplicate
AfterFetcher Pool receives URL to fetch
Fetcher Pool

Fetch URL content

Fetcher Pool downloads the web page content from the internet

Protocol:HTTP
BeforeFetcher Pool has URL to fetch
AfterFetcher Pool retrieves page content and sends results downstream
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────┐
│ URL Frontier  │
└───────┬───────┘
        │ Kafka
        ▼
┌───────────────┐
│ Scheduler Node│
└───────┬───────┘
        │ RPC
        ▼
┌───────────────┐
│Politeness     │
│Enforcer       │
└───────┬───────┘
        │ REST API
        ▼
┌───────────────┐
│Duplicate      │
│Detection DB   │
└───────┬───────┘
        │ DB Query
        ▼
┌───────────────┐
│ Fetcher Pool  │
└───────────────┘
Design Rationale

This web crawler architecture processes URLs through a vertical pipeline starting at the URL Frontier queue, which holds URLs prioritized for crawling. The Scheduler Node consumes URLs from the Frontier and consults the Politeness Enforcer to ensure crawl delays per domain are respected, preventing server overload. After politeness approval, the Scheduler queries the Duplicate Detection DB to avoid re-fetching already crawled URLs. Only URLs passing these checks are dispatched to the Fetcher Pool, which performs the actual HTTP fetch. This design cleanly separates concerns: scheduling, politeness, duplicate detection, and fetching, enabling scalable and polite crawling.

Primary Focus

✓

Practice

(1/5)
1. In a primary-replica replication system, at step 4 the replica receives a batch of write-ahead log entries from the primary. What is the replica's exact state after applying these entries?
easy
A. Replica marks entries as committed and acknowledges primary before applying
B. Replica applies entries to its data store and then acknowledges primary
C. Replica applies entries and immediately serves reads from them before acknowledgment
D. Replica buffers entries without applying until primary confirms commit

Solution

  1. Step 1: Trace replica behavior at step 4

    Replica applies WAL entries to its local store to catch up with primary.

  2. Step 2: Understand acknowledgment timing

    Replica acknowledges primary only after applying entries to ensure durability and consistency.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Replica applies then acknowledges -> ensures data durability [check]
Hint: Apply WAL entries before ack -> durability ensured [check]
Common Mistakes:
2. A real-time location tracking system must efficiently query nearby devices within a geographic radius with low latency. Which component is best suited to handle these geo-spatial queries?
easy
A. A relational database with spatial extensions (e.g., PostGIS) for all queries
B. A geo-indexing service like a geospatially-enabled NoSQL store (e.g., Redis GEO or Elasticsearch)
C. A message queue to buffer location updates before querying
D. A CDN to cache location data for faster access

Solution

  1. Step 1: Identify geo-indexing role

    Relational DBs with spatial extensions provide accuracy but have higher latency and poor horizontal scaling for frequent geo queries.

  2. Step 2: Evaluate alternatives

    Message queues buffer data but do not support geo queries. CDNs cache static content, not dynamic geo data.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Geo-indexing NoSQL stores optimize radius queries with low latency [check]
Hint: Geo-indexing NoSQL stores excel at fast radius queries [check]
Common Mistakes:
3. In a system generating IDs using a custom ID service, at step 3 the ID generator assigns a timestamp-based prefix before appending a sequence number. What happens if the sequence number reaches its max value before the timestamp changes?
easy
A. The ID generator switches to UUID generation to avoid sequence overflow
B. The ID generator blocks until the timestamp increments to avoid collisions
C. The ID generator resets the sequence number to zero immediately, risking duplicate IDs
D. The ID generator skips the timestamp and uses only the sequence number

Solution

  1. Step 1: Trace ID generation at step 3

    The timestamp prefix combined with a sequence number ensures uniqueness within the same timestamp.

  2. Step 2: Understand sequence number overflow handling

    When the sequence number maxes out, the generator must wait for the timestamp to increment to avoid duplicates, thus blocking new ID generation temporarily.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Sequence overflow -> block until timestamp changes [check]
Hint: Sequence maxed -> wait for timestamp tick [check]
Common Mistakes:
4. If the origin server becomes unreachable, what exactly breaks in a CDN using origin pull with TTL caching?
medium
A. Edge nodes continue serving stale cached content indefinitely without errors
B. Edge nodes serve stale content until TTL expires, then return errors for new requests
C. Edge nodes immediately return errors for all requests without cached content
D. Edge nodes switch to push mode automatically to fetch content from other origins

Solution

  1. Step 1: Understand origin pull with TTL caching failure

    When origin is unreachable, edge nodes serve cached content until TTL expires.

  2. Step 2: Identify behavior after TTL expiry

    After TTL expires, edge nodes cannot refresh content and must return errors for new requests.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Origin down + expired TTL -> edge nodes return errors [check]
Hint: Expired TTL + origin down -> errors returned [check]
Common Mistakes:
5. If the global BGP route monitoring system is disabled during a misconfiguration event, what is the most likely failure observed in the network recovery process?
medium
A. Routers automatically revert to default static routes, causing traffic blackholing
B. Network devices silently drop BGP updates, leading to partial route visibility
C. Operators receive delayed alerts, causing prolonged outage and inconsistent routing states
D. Clients experience immediate failover to backup ISPs without disruption

Solution

  1. Step 1: Role of BGP monitoring

    BGP monitoring detects anomalies and route changes; without it, routers may silently drop updates due to misconfigurations.

  2. Step 2: Eliminate other options

    Delayed alerts (A) affect operators but don't cause silent drops; routers don't revert to static routes automatically (B); immediate client failover (D) unlikely without proper routing.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Without monitoring, routers drop BGP updates silently causing partial visibility [check]
Hint: No monitoring -> silent BGP update drops, partial routes [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard