Robots.txt configuration in SEO Fundamentals - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of a robots.txt file?

A robots.txt file tells web robots (like search engine crawlers) which parts of a website they can or cannot visit. It helps control what content is indexed.

beginner

What does the User-agent directive specify in a robots.txt file?

The User-agent directive specifies which web robot the following rules apply to. For example, User-agent: Googlebot targets Google's crawler.

beginner

How do you block all web crawlers from accessing your entire website using robots.txt?

You write:<br>User-agent: *<br>Disallow: /<br>This tells all robots not to visit any pages on the site.

beginner

What does the Disallow directive do in a robots.txt file?

The Disallow directive tells the specified user-agent which paths or pages it should NOT crawl.

intermediate

Can robots.txt prevent a page from being indexed if other sites link to it?

No. robots.txt only controls crawling. If other sites link to a page, search engines might still index its URL without content.

What does User-agent: * mean in a robots.txt file?

A It applies rules only to Googlebot

B It allows all users to access the website

C It blocks all users from the website

D It applies rules to all web crawlers

User-agent: * means the rules apply to all web crawlers or robots.

How do you allow all web crawlers to access your entire website?

A User-agent: *<br>Disallow: /

B User-agent: *<br>Disallow:

C User-agent: Googlebot<br>Disallow: /

D User-agent: *<br>Allow: /private

Leaving Disallow: empty means no pages are blocked, so all are allowed.

Which directive blocks a specific folder from being crawled?

A Disallow: /folder/

B Allow: /folder/

C User-agent: /folder/

D Block: /folder/

Disallow: /folder/ tells robots not to crawl that folder.

If a page is blocked by robots.txt, can it still appear in search results?

A Only if the page is on the homepage

B No, it will never appear

C Yes, if other sites link to it

D Only if the page has a sitemap

Blocking crawling does not guarantee no indexing if other sites link to the page.

Where should the robots.txt file be placed on a website?

A In the root directory of the website

B In the images folder

C In the CSS folder

D Anywhere on the website

The robots.txt file must be in the root directory to be found by crawlers.

Explain how a robots.txt file controls web crawler access to a website.

Think about how you tell robots where they can and cannot go.

Describe a scenario where blocking a page with robots.txt might not prevent it from appearing in search results.

Consider what happens if other sites link to a blocked page.