Robots.txt configuration in NextJS - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to import the package for robots.txt generation in Next.js.

NextJS

import [1] from 'nextjs-robots';

Drag options to blanks, or click blank then click option'

AgenerateRobots

Brobots

CcreateRobotsTxt

DRobotsTxt

Attempts:

3 left

💡 Hint

Common Mistakes

Using lowercase or incorrect import names.

Trying to import from a wrong package.

The package exports RobotsTxt component to generate robots.txt in Next.js.

2fill in blank

medium

Complete the code to define a basic robots.txt rule allowing all user agents.

NextJS

export const robotsTxtOptions = {
  policies: [
    { userAgent: '*', [1]: true }
  ]
};

Drag options to blanks, or click blank then click option'

Adisallow

Bcrawl

Callow

Dpermit

Attempts:

3 left

💡 Hint

Common Mistakes

Using disallow instead of allow for permitting access.

Using non-existent properties like crawl or permit.

The allow property set to true means all user agents can crawl the site.

3fill in blank

hard

Fix the error in the robots.txt options to block a specific path '/admin'.

NextJS

export const robotsTxtOptions = {
  policies: [
    { userAgent: '*', disallow: [1] }
  ]
};

Drag options to blanks, or click blank then click option'

A'/admin'

B'admin/'

C/admin

Dadmin

Attempts:

3 left

💡 Hint

Common Mistakes

Omitting the leading slash.

Using quotes incorrectly or missing quotes.

The path must be a string starting with a slash, so '/admin' is correct.

4fill in blank

hard

Fill both blanks to add sitemap URL and host in robots.txt options.

NextJS

export const robotsTxtOptions = {
  sitemap: '[1]',
  host: '[2]'
};

Drag options to blanks, or click blank then click option'

Ahttps://example.com/sitemap.xml

Bhttps://example.com

Chttp://example.com

Dhttps://example.org

Attempts:

3 left

💡 Hint

Common Mistakes

Mixing host and sitemap URLs.

Using HTTP instead of HTTPS if site uses HTTPS.

The sitemap should point to the sitemap XML URL, and host is the main site URL.

5fill in blank

hard

Fill all three blanks to create a robots.txt component with options and export it as default.

NextJS

import [1] from 'nextjs-robots';

const robots = () => {
  return <[2] options={robotsTxtOptions} />;
};

export default [3];

Drag options to blanks, or click blank then click option'

ARobotsTxt

BRobotsTxtComponent

Crobots

DRobots

Attempts:

3 left

💡 Hint

Common Mistakes

Using different names for import and JSX tag.

Exporting the wrong identifier.

Import RobotsTxt, use it as a component, and export the function robots as default.

Practice

1. What is the main purpose of a robots.txt file in a Next.js project?

easy

A. To manage server-side rendering

B. To tell search engines which pages to crawl or not crawl

C. To store user data securely

D. To style the website pages

Solution

1. Step 1: Understand the role of robots.txt

   The robots.txt file is used to communicate with search engines about which parts of the website they can access.

2. Step 2: Identify its purpose in Next.js

   In Next.js, it is used the same way as in other web projects: to allow or block search engine crawlers from certain pages.

3. Final Answer:

   To tell search engines which pages to crawl or not crawl -> Option B

4. Quick Check:

   robots.txt controls crawler access = D [OK]

Hint: robots.txt controls search engine access [OK]

Common Mistakes:

• Confusing robots.txt with CSS or styling files
• Thinking robots.txt stores user data
• Assuming robots.txt manages rendering

2. Which of the following is the correct way to serve a robots.txt file using an API route in Next.js?

easy

A. export default function handler(req, res) { res.setHeader('Content-Type', 'text/html'); res.send('User-agent: *Disallow: /private'); }

B. export default function handler(req, res) { res.setHeader('Content-Type', 'application/json'); res.send({ 'User-agent': '*', 'Disallow': '/private' }); }

C. export default function handler(req, res) { res.setHeader('Content-Type', 'text/plain'); res.send('User-agent: *\nDisallow: /private'); }

D. export default function handler(req, res) { res.setHeader('Content-Type', 'application/javascript'); res.send('User-agent: *\nDisallow: /private'); }

Solution

1. Step 1: Check the content type for robots.txt

   The robots.txt file must be served as plain text, so the header should be text/plain.

2. Step 2: Verify the response content format

   The content must be plain text lines, not JSON or HTML, so the string with newlines is correct.

3. Final Answer:

   export default function handler(req, res) { res.setHeader('Content-Type', 'text/plain'); res.send('User-agent: *\nDisallow: /private'); } -> Option C

4. Quick Check:

   robots.txt served as plain text = A [OK]

Hint: robots.txt must be served as plain text [OK]

Common Mistakes:

• Using JSON or HTML content type instead of text/plain
• Sending JSON object instead of plain text string
• Forgetting to set Content-Type header

3. Given this Next.js API route code for robots.txt:

export default function handler(req, res) {
  res.setHeader('Content-Type', 'text/plain');
  res.send(`User-agent: *\nDisallow: /admin\nAllow: /public`);
}

What will search engines do when they read this robots.txt?

medium

A. They will block crawling of /admin but allow /public and other pages

B. They will allow crawling of all pages including /admin

C. They will block crawling of all pages including /public

D. They will ignore the file because of syntax errors

Solution

1. Step 1: Read the robots.txt rules

   The file says User-agent: * (all bots), Disallow: /admin (block /admin), and Allow: /public (allow /public).

2. Step 2: Understand the effect on crawling

   Search engines will not crawl /admin but will crawl /public and other pages not disallowed.

3. Final Answer:

   They will block crawling of /admin but allow /public and other pages -> Option A

4. Quick Check:

   Disallow /admin, Allow /public = C [OK]

Hint: Disallow blocks, Allow permits specific paths [OK]

Common Mistakes:

• Thinking Allow overrides Disallow globally
• Assuming all pages are blocked
• Confusing syntax causing ignored file

4. You wrote this Next.js API route for robots.txt:

export default function handler(req, res) {
  res.setHeader('Content-Type', 'text/plain')
  res.send('User-agent: *\nDisallow: /secret')
}

But when you visit /api/robots.txt, the browser downloads a file instead of showing text. What is the likely cause?

medium

A. The API route is correct; browser behavior depends on browser settings

B. The Content-Type header should be 'text/html' not 'text/plain'

C. The browser treats text/plain as a download, so you must add Content-Disposition header

D. Missing semicolons cause syntax error

Solution

1. Step 1: Check the code syntax

   The code is valid JavaScript; semicolons are optional in this context and do not cause errors.

2. Step 2: Understand browser behavior for text/plain

   Browsers may display or download text/plain files depending on settings; this is normal and not an error in the API.

3. Final Answer:

   The API route is correct; browser behavior depends on browser settings -> Option A

4. Quick Check:

   text/plain may download = A [OK]

Hint: text/plain may download depending on browser [OK]

Common Mistakes:

• Assuming missing semicolons cause errors
• Changing Content-Type to text/html incorrectly
• Adding unnecessary headers to force display

5. You want to block all search engines from crawling any page except the homepage and the /blog section in your Next.js app. Which robots.txt content served via API route achieves this correctly?

hard

A. User-agent: *\nDisallow: /\nAllow: /\nAllow: /blog/

B. User-agent: *\nDisallow: /\nAllow: /\nAllow: /blog

C. User-agent: *\nDisallow: /\nAllow: /index.html\nAllow: /blog/

D. User-agent: *\nDisallow: /\nAllow: /$\nAllow: /blog/

Solution

1. Step 1: Understand the goal

   Block all pages except homepage and /blog section. Homepage is usually root path '/' only.

2. Step 2: Use correct syntax for homepage allowance

   To allow only the homepage, use Allow: /$ which matches exactly the root URL, not all paths starting with '/'.

3. Step 3: Allow /blog/ section

   Allowing /blog/ lets crawlers access all pages under /blog.

4. Final Answer:

   User-agent: *\nDisallow: /\nAllow: /$\nAllow: /blog/ -> Option D

5. Quick Check:

   Disallow all except root and /blog = B [OK]

Hint: Use Allow: /$ to allow only homepage [OK]

Common Mistakes:

• Allowing '/' allows all pages, not just homepage
• Using Allow: /blog without trailing slash misses subpages
• Confusing homepage URL with /index.html