How Google discovers pages (crawling) in SEO Fundamentals - Try It Yourself

How Google Discovers Pages (Crawling)

📖 Scenario: You are learning how Google finds new web pages on the internet. This process is called crawling. Understanding crawling helps website owners make their pages easy to find.

🎯 Goal: Build a simple step-by-step outline that shows how Google discovers pages by starting from known URLs and following links.

📋 What You'll Learn

Create a list called seed_urls with three example starting web addresses

Create a variable called max_pages to limit how many pages Google will try to find

Write a loop using for url in seed_urls to simulate visiting each starting page

Add a final step that shows adding a new URL to the list to simulate discovering a new page

💡 Why This Matters

🌍 Real World

Understanding crawling helps website owners make sure their pages are found by Google and appear in search results.

💼 Career

SEO specialists and web developers use knowledge of crawling to improve website visibility and optimize site structure.

1

Create the starting URLs list

Create a list called seed_urls with these exact URLs: 'https://example.com', 'https://example.org', and 'https://example.net'.

SEO Fundamentals

# Create the list of starting URLs
# Your code here

Need a hint?

Use square brackets [] to create a list and separate URLs with commas.

2

Set the maximum pages limit

Create a variable called max_pages and set it to the number 10 to limit how many pages Google will crawl.

SEO Fundamentals

seed_urls = ['https://example.com', 'https://example.org', 'https://example.net']
# Set the max_pages limit
# Your code here

Need a hint?

Just assign the number 10 to the variable max_pages.

3

Simulate visiting each starting URL

Write a for loop using for url in seed_urls to simulate Google visiting each URL in the list.

SEO Fundamentals

seed_urls = ['https://example.com', 'https://example.org', 'https://example.net']
max_pages = 10

# Loop through each URL in seed_urls
# Your code here

Need a hint?

Use a for loop with the variable name url to go through seed_urls.

4

Add a new discovered URL

Inside the loop, add a new URL 'https://example.com/about' to the seed_urls list to simulate Google discovering a new page.

SEO Fundamentals

seed_urls = ['https://example.com', 'https://example.org', 'https://example.net']
max_pages = 10

for url in seed_urls:
    # Add a new discovered URL
    # Your code here

Need a hint?

Use the append() method on seed_urls to add the new URL.