Robots.txt Generator & Validator

The Robots Exclusion Protocol (REP), formalized in RFC 9309, is the universal standard that webmasters use to communicate with automated web crawlers, search engine spiders (such as Googlebot and Bingbot), and AI scraping agents (such as GPTBot and ClaudeBot).

A robots.txt file is a plain text file that must reside at the exact root of your web server (e.g. https://example.com/robots.txt). When a crawler visits your site, it initiates a GET /robots.txt request before requesting any other webpage or media asset. The instructions inside indicate which URLs or directories the crawler is permitted (Allow) or forbidden (Disallow) from requesting.

• Domain Root Placement: The file must always be placed at the domain root. Subdirectory files (like example.com/blog/robots.txt) are ignored by all major search engines.
• Case Sensitivity: Directives (like User-agent: and Disallow:) are case-insensitive, but directory paths are strictly case-sensitive. Disallow: /admin/ does not block /Admin/.
• Wildcard Matching: Modern REP parsers support wildcard matching (*) for sequence matching and end-of-string anchors ($) to target specific file extensions like /*.pdf$.

Configuring robots.txt incorrectly can lead to catastrophic organic search drops or unintended sensitive data leaks. Key pitfalls to avoid include:

1. Disallow: / vs Disallow: Disallow: / blocks crawlers from accessing your entire website (common on staging environments). Conversely, an empty Disallow: directive allows crawlers complete, unrestricted access to the entire site.
2. Robots.txt Does NOT Prevent Indexing: Believing that Disallow: /private/ prevents a page from being indexed in Google Search is one of the most common SEO mistakes. Robots.txt prevents crawling (downloading the page content), not indexing. If external sites link to the disallowed URL, Google can index the URL as a bare link without snippet text. To guarantee a page is never indexed, allow crawling and serve a <meta name="robots" content="noindex"> tag or an X-Robots-Tag: noindex HTTP response header.
3. Blocking CSS and JavaScript Assets: Never disallow /css/, /js/, or font directories. Googlebot requires full layout rendering capabilities to verify mobile-friendliness and calculate Core Web Vitals.
4. Crawl-Delay Directive Inconsistencies: While Bingbot, Yandex, and Baidu recognize the Crawl-delay directive (measured in seconds), Googlebot ignores Crawl-delay completely in favor of autonomous algorithmic crawl-rate controls in Google Search Console.

This generator and validator is powered by omniseo-core, our high-performance, zero-dependency open-source library for SEO calculation and validation.

You can integrate the exact same parsing, serialization, and linting logic directly into your own CI/CD pipelines, Next.js build steps, or automated site auditing tools:

npm install omniseo-core

Key functions include:

• validateRobotsTxt(raw: string): Validates syntax, catches missing User-agents, checks protocol on Sitemaps, flags unencoded path characters, and outputs line-numbered diagnostics.
• parseRobotsTxt(raw: string): Deserializes raw text into structured TypeScript RobotsConfig objects.
• generateRobotsTxt(config: RobotsConfig): Serializes configuration objects into clean, standard-compliant robots.txt text.