How to Design a Scalable Web Crawler: Key Concepts and Example

To design a web crawler, build a system that starts from seed URLs, fetches web pages using HTTP requests, extracts links, and stores data. Use queues to manage URLs, parsers to extract content, and rate limiting to avoid overloading servers.

Syntax

A basic web crawler follows these steps:

• Seed URLs: Start with a list of initial URLs.
• Fetch: Use HTTP requests to download page content.
• Parse: Extract useful data and links from the page.
• Queue: Add new links to a queue for future crawling.
• Store: Save extracted data in a database or file.
• Repeat: Continue until the queue is empty or a limit is reached.

python

class WebCrawler:
    def __init__(self, seeds):
        self.queue = seeds  # URLs to visit
        self.visited = set()  # Track visited URLs

    def fetch(self, url):
        # Download page content
        pass

    def parse(self, content):
        # Extract data and links
        return [], []  # data, links

    def crawl(self):
        while self.queue:
            url = self.queue.pop(0)
            if url in self.visited:
                continue
            content = self.fetch(url)
            data, links = self.parse(content)
            self.visited.add(url)
            self.queue.extend(links)
            self.store(data)

    def store(self, data):
        # Save extracted data
        pass

Example

This example shows a simple crawler that fetches a page, extracts all links, and prints them. It uses Python's requests and BeautifulSoup libraries.

python

import requests
from bs4 import BeautifulSoup

class SimpleCrawler:
    def __init__(self, seeds):
        self.queue = seeds
        self.visited = set()

    def fetch(self, url):
        try:
            response = requests.get(url, timeout=5)
            if response.status_code == 200:
                return response.text
        except requests.RequestException:
            return None

    def parse(self, content):
        soup = BeautifulSoup(content, 'html.parser')
        links = []
        for a_tag in soup.find_all('a', href=True):
            href = a_tag['href']
            if href.startswith('http'):
                links.append(href)
        return links

    def crawl(self):
        while self.queue:
            url = self.queue.pop(0)
            if url in self.visited:
                continue
            print(f'Crawling: {url}')
            content = self.fetch(url)
            if content:
                links = self.parse(content)
                self.visited.add(url)
                self.queue.extend([link for link in links if link not in self.visited])

if __name__ == '__main__':
    seeds = ['https://example.com']
    crawler = SimpleCrawler(seeds)
    crawler.crawl()

Output

Crawling: https://example.com
Crawling: https://www.iana.org/domains/example

Common Pitfalls

• Infinite loops: Crawlers can revisit the same URLs endlessly without tracking visited links.
• Overloading servers: Sending too many requests too fast can cause bans or slowdowns; use rate limiting.
• Ignoring robots.txt: Not respecting robots.txt rules can lead to legal or ethical issues.
• Poor URL normalization: Different URL forms may point to the same page; normalize URLs to avoid duplicates.
• Not handling errors: Network failures or invalid pages must be handled gracefully.

python

## Wrong: No visited check, causes infinite loop
queue = ['https://example.com']
while queue:
    url = queue.pop(0)
    print(f'Fetching {url}')
    # Adds same URL again
    queue.append(url)

## Right: Track visited URLs
queue = ['https://example.com']
visited = set()
while queue:
    url = queue.pop(0)
    if url in visited:
        continue
    print(f'Fetching {url}')
    visited.add(url)
    # Add new URLs only if not visited
    # queue.extend(new_urls)

Quick Reference

• Start with a small set of seed URLs.
• Use queues to manage URLs to visit.
• Track visited URLs to avoid repeats.
• Respect robots.txt and rate limits.
• Parse pages to extract data and new links.
• Store data efficiently for later use.

Key Takeaways

Use queues and visited sets to manage URLs and avoid infinite loops.

Respect website rules like robots.txt and apply rate limiting to be polite.

Parse HTML carefully to extract useful data and new links.

Handle network errors and invalid pages gracefully.

Store extracted data in a structured way for easy access.