Robots.txt configuration in NextJS - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ component_behavior

intermediate

What is the effect of this robots.txt configuration in Next.js?

Given this robots.txt content served in a Next.js app, what will search engines be allowed to crawl?

NextJS

User-agent: *
Disallow: /admin
Allow: /admin/public
Sitemap: https://example.com/sitemap.xml

A Search engines can crawl all pages except those under /admin except /admin/public which is allowed.

B Search engines cannot crawl any pages under /admin including /admin/public.

C Search engines can crawl all pages including /admin and /admin/public.

D Search engines are blocked from crawling the entire site.

💡 Hint

Remember that 'Allow' can override 'Disallow' for specific paths.

The Disallow: /admin blocks crawling of the /admin directory, but Allow: /admin/public explicitly permits crawling of the /admin/public subdirectory. The User-agent: * applies this rule to all search engines.

📝 Syntax

intermediate

Which robots.txt snippet is syntactically correct for Next.js static export?

Choose the valid robots.txt content that Next.js can serve without syntax errors.

A

User-agent: *
Disallow: /private
Allow: /public
Sitemap https://example.com/sitemap.xml
Extra: value

B

User-agent *
Disallow /private
Allow /public
Sitemap https://example.com/sitemap.xml

C

User-agent: *
Disallow: /private
Allow: /public
Sitemap: https://example.com/sitemap.xml

D

User-agent: *
Disallow: /private
Allow: /public
Sitemap: https://example.com/sitemap.xml
User-agent: /bot

💡 Hint

Check for correct use of colons and standard directives.

Option C uses correct syntax with colons and standard directives. Option C misses colons, Option C has an invalid 'Extra' directive, and Option C has an invalid 'User-agent' value.

🔧 Debug

advanced

Why does this Next.js robots.txt not block /secret as intended?

This robots.txt is served but /secret is still indexed by search engines. What is the issue?

NextJS

User-agent: *
Disallow: secret

A The 'User-agent' directive is incorrect and should be 'User-agent: *'.

B The path 'secret' is missing a leading slash, so it doesn't match '/secret'.

C The 'Disallow' directive should be 'Disallow: /secret/*' to block all subpaths.

D The robots.txt file must be named 'robotsjs.txt' in Next.js.

💡 Hint

Paths in robots.txt must start with a slash to match URLs correctly.

Robots.txt paths must start with a slash. Without it, 'secret' does not match '/secret' URLs, so crawling is not blocked.

🧠 Conceptual

advanced

What is the purpose of including a Sitemap directive in robots.txt in Next.js?

Why should you add a Sitemap URL in your robots.txt file when deploying a Next.js site?

A It helps search engines find and index all pages efficiently by pointing to the sitemap location.

B It blocks search engines from crawling the sitemap file itself.

C It automatically generates the sitemap for the Next.js site.

D It prevents duplicate content issues by redirecting crawlers.

💡 Hint

Think about how search engines discover pages on your site.

The Sitemap directive tells search engines where to find the sitemap XML file, which lists all important URLs. This improves crawling efficiency and indexing.

❓ state_output

expert

What is the output behavior when Next.js serves this robots.txt file dynamically?

Consider a Next.js API route serving robots.txt with this code snippet: ```js export default function handler(req, res) { res.setHeader('Content-Type', 'text/plain'); res.write('User-agent: *\nDisallow: /private'); res.end(); } ``` What will a search engine see when requesting /robots.txt?

NextJS

export default function handler(req, res) {
  res.setHeader('Content-Type', 'text/plain');
  res.write('User-agent: *\nDisallow: /private');
  res.end();
}

A The search engine receives an HTML page instead of robots.txt content.

B The robots.txt content is served but with incorrect Content-Type causing it to be ignored.

C The server returns a 404 Not Found error for /robots.txt.

D The search engine receives a valid robots.txt blocking /private directory.

💡 Hint

Check the Content-Type header and response body.

The handler sets the Content-Type to 'text/plain' and writes valid robots.txt content, so search engines will correctly interpret it and block /private.