Next.js Robots.txt Generator & app/robots.ts Validator

Modern Next.js applications (Next.js 13.3+, Next.js 14, and Next.js 15+) replace legacy static public/robots.txt files with special Metadata Route Handlers (app/robots.ts or app/robots.js).

Why Use app/robots.ts Over public/robots.txt?

Static files in the public/ folder cannot react to environment variables or dynamic deployment URLs. In contrast, app/robots.ts gives you full programmatic control:

• Environment Branch Protection: Automatically disallow search engines on Vercel Preview or staging branches (VERCEL_ENV !== 'production') while keeping production crawlable.
• Dynamic Sitemaps: Automatically concatenate multi-language or multi-section sitemap URLs based on runtime configuration.
• Strict Type Safety: TypeScript validates that your rules, user agents, allow paths, and crawl delays match Next.js MetadataRoute.Robots specifications.

Production Next.js app/robots.ts Code Structure

Below is the recommended production implementation for Next.js App Router:

// app/robots.ts (Next.js App Router)
import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  const isProduction = process.env.NEXT_PUBLIC_SITE_ENV === 'production';
  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://yourdomain.com';

  // Disallow all crawlers on Staging / Preview deployments
  if (!isProduction) {
    return {
      rules: {
        userAgent: '*',
        disallow: '/',
      },
    };
  }

  // Production: Allow standard search engines + block AI model scrapers
  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/api/', '/admin/', '/private/', '/_next/'],
      },
      {
        userAgent: ['GPTBot', 'ClaudeBot', 'Google-Extended', 'CCBot', 'Bytespider'],
        disallow: '/',
      },
    ],
    sitemap: `${baseUrl}/sitemap.xml`,
    host: baseUrl,
  };
}

// app/robots.ts (Next.js App Router)
import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  const isProduction = process.env.NEXT_PUBLIC_APP_ENV === 'production';
  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://yourdomain.com';

  if (!isProduction) {
    return {
      rules: {
        userAgent: '*',
        disallow: '/',
      },
    };
  }

  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/api/', '/admin/', '/private/', '/_next/'],
      },
      {
        userAgent: ['GPTBot', 'ClaudeBot', 'Google-Extended', 'CCBot', 'Bytespider'],
        disallow: '/',
      },
    ],
    sitemap: `${baseUrl}/sitemap.xml`,
    host: baseUrl,
  };
}