NextJSframework~30 mins

Robots.txt configuration in NextJS - Mini Project: Build & Apply

Choose your learning style9 modes available

Learn Why Deep Visual Try Challenge Project Recall Perf

Robots.txt Configuration in Next.js

📖 Scenario: You are building a website using Next.js. You want to control how search engines find and index your pages. To do this, you will create a robots.txt file that tells search engines which pages to crawl and which to avoid.

🎯 Goal: Create a robots.txt file in a Next.js project that allows all search engines to crawl the homepage and blocks them from crawling the /admin page.

📋 What You'll Learn

Create a robots.txt file with the exact content to allow all user agents to crawl the homepage

Disallow crawling of the /admin path

Serve the robots.txt file correctly from the Next.js public folder

Verify the robots.txt file is accessible at /robots.txt in the browser

💡 Why This Matters

🌍 Real World

Websites use <code>robots.txt</code> files to control how search engines crawl and index their pages. This helps protect private areas and improve SEO.

💼 Career

Knowing how to configure <code>robots.txt</code> is important for web developers and SEO specialists to manage site visibility and search engine behavior.

Progress0 / 4 steps

Create the robots.txt file

Create a file named robots.txt inside the public folder of your Next.js project. Add these exact lines to the file:
User-agent: *
Allow: /

NextJS

# Create the robots.txt file in the public folder with the required lines
# Your code here

Need a hint?

The robots.txt file must be placed inside the public folder so Next.js can serve it automatically.

Add disallow rule for /admin path

Add a line to the existing robots.txt file to disallow all user agents from crawling the /admin path. The line should be exactly Disallow: /admin.

NextJS

User-agent: *
Allow: /
# Add the disallow rule for /admin below
# Your code here

Need a hint?

The Disallow directive tells search engines not to crawl the specified path.

Verify robots.txt is served by Next.js

Run your Next.js development server and verify that the robots.txt file is accessible by visiting http://localhost:3000/robots.txt in your browser.

NextJS

User-agent: *
Allow: /
Disallow: /admin
# Run the Next.js server and check the URL /robots.txt
# Your code here

Need a hint?

Next.js automatically serves files placed in the public folder at the root URL path.

Add meta tag for robots in the homepage

Open the pages/index.js file and add a <meta name="robots" content="index, follow" /> tag inside the <head> section to explicitly allow search engines to index and follow links on the homepage.

NextJS

import Head from 'next/head';

export default function Home() {
  return (
    
      
        {/* Add the meta robots tag below */}
        {/* Your code here */}
      
      
        Welcome to the homepage
      
    
  );
}

Need a hint?

The meta tag inside the <head> helps search engines understand how to treat this page.