Robots.txt configuration in NextJS

Introduction

The robots.txt file tells search engines which pages to look at or ignore on your website. It helps control what shows up in search results.

You want to hide private pages from search engines.

You want to prevent search engines from indexing duplicate content.

You want to allow all search engines to crawl your entire site.

You want to block specific bots from accessing your site.

You want to improve SEO by guiding search engines.

Syntax

NextJS

User-agent: [name]
Disallow: [path]
Allow: [path]
Sitemap: [url]

User-agent specifies which search engine bot the rule applies to.

Disallow tells bots not to visit certain pages or folders.

Examples

Blocks all bots from accessing the /private/ folder.

NextJS

User-agent: *
Disallow: /private/

Blocks only Google's bot from the /no-google/ folder.

NextJS

User-agent: Googlebot
Disallow: /no-google/

Blocks everything except the /public/ folder for all bots.

NextJS

User-agent: *
Allow: /public/
Disallow: /

Sample Program

This Next.js route returns a robots.txt file that blocks all bots from the /admin/ folder but allows crawling everywhere else. It also provides the sitemap URL.

NextJS

import { NextResponse } from 'next/server';

export function GET() {
  const robots = `User-agent: *
Disallow: /admin/
Allow: /
Sitemap: https://example.com/sitemap.xml`;
  return new NextResponse(robots, {
    status: 200,
    headers: {
      'Content-Type': 'text/plain'
    }
  });
}

Important Notes

Place the robots.txt file at the root of your website for search engines to find it.

Use Next.js API routes or middleware to serve robots.txt dynamically if needed.

Test your robots.txt using Google Search Console to ensure it works as expected.

Summary

Robots.txt controls what search engines can see on your site.

You can block or allow specific pages or bots.

In Next.js, you can serve robots.txt using an API route returning plain text.