Robots.txt configuration in NextJS - Step-by-Step Execution

Concept Flow - Robots.txt configuration

Start

↓

Create robots.txt file

↓

Add rules: User-agent, Allow/Disallow

↓

Place file in public folder

↓

Build and deploy Next.js app

↓

Search engines request /robots.txt

↓

Serve robots.txt content

↓

Search engines follow rules

↓

End

The flow shows creating a robots.txt file with rules, placing it in the public folder, deploying the app, and search engines reading and following the rules.

Execution Sample

NextJS

User-agent: *
Disallow: /admin
Allow: /

# Place this file in /public folder

This robots.txt file blocks all search engines from /admin but allows crawling other pages.

Execution Table

Step	Action	File Location	Content Added	Effect
1	Create robots.txt file	/public	User-agent: *	Targets all search engines
2	Add Disallow rule	/public	Disallow: /admin	Blocks /admin from crawling
3	Add Allow rule	/public	Allow: /	Allows crawling all other pages
4	Deploy Next.js app	N/A	robots.txt served at /robots.txt	File accessible to crawlers
5	Search engine requests /robots.txt	N/A	robots.txt content served	Crawler reads rules
6	Crawler follows rules	N/A	Blocks /admin, allows others	Site indexed accordingly
7	End	N/A	N/A	Process complete

💡 robots.txt served and crawlers follow rules, stopping at step 6

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5	Final
robots.txt content	empty	User-agent: *	User-agent: * Disallow: /admin	User-agent: * Disallow: /admin Allow: /	File placed in /public	File served at /robots.txt	Rules read and applied by crawler

Key Moments - 3 Insights

Why must robots.txt be placed in the /public folder in Next.js?

Because Next.js serves static files from the /public folder at the root URL, placing robots.txt there makes it accessible at /robots.txt for crawlers, as shown in execution_table step 4.

What happens if you forget to add 'User-agent: *' in robots.txt?

Without 'User-agent: *', the rules may not apply to all crawlers. The execution_table step 1 shows this line targets all search engines, so missing it means rules might be ignored.

Can you block only one page or folder using robots.txt?

Yes, by adding 'Disallow: /folder' or 'Disallow: /page', like in step 2 where /admin is blocked, while others remain allowed.

Visual Quiz - 3 Questions

Look at the execution_table at step 3, what rule is added to allow crawling?

A User-agent: *

B Disallow: /admin

C Allow: /

D Disallow: /

💡 Hint

Check the 'Content Added' column at step 3 in execution_table

At which step does the robots.txt file become accessible to search engines?

A Step 4

B Step 2

C Step 5

D Step 6

💡 Hint

Look for when the file is served at /robots.txt in execution_table

If you remove 'Disallow: /admin', what changes in the crawler behavior?

A Crawler blocks /admin

B Crawler allows /admin

C Crawler blocks all pages

D Crawler ignores robots.txt

💡 Hint

Refer to variable_tracker and execution_table step 2 and 6 about disallow rules

Concept Snapshot

robots.txt is a text file placed in the /public folder in Next.js.
It tells search engines which pages or folders to allow or block.
Syntax uses 'User-agent', 'Disallow', and 'Allow' lines.
Search engines read /robots.txt at site root and follow rules.
Always test your robots.txt by accessing /robots.txt URL.
Blocking sensitive pages improves site privacy and SEO control.

Full Transcript

Robots.txt configuration in Next.js involves creating a plain text file named robots.txt with rules for search engines. The file must be placed in the /public folder so it is served at the root URL /robots.txt. The file contains lines like 'User-agent: *' to target all crawlers, 'Disallow: /admin' to block crawling of the /admin folder, and 'Allow: /' to permit crawling of other pages. When the Next.js app is deployed, search engines request /robots.txt and read these rules to decide which pages to index. This process helps control what parts of the site appear in search results. Beginners often wonder why the file must be in /public and what happens if rules are missing. The execution table shows step-by-step how the file is created, deployed, served, and used by crawlers. The variable tracker shows how the content of robots.txt builds up over steps. Quizzes test understanding of when rules are added, when the file is accessible, and effects of removing rules. The snapshot summarizes key points for quick reference.