Shopify Robots.txt Generator & Liquid Validator

Shopify manages crawler indexation through a native templating system introduced in Online Store 2.0. Rather than uploading a static robots.txt file to the server root (which Shopify's hosted architecture does not allow), Shopify renders the file dynamically via templates/robots.txt.liquid.

The Native Liquid robots.default_groups Architecture

In standard Shopify themes (such as Dawn, Sense, Spotlight, and Prestige), the default templates/robots.txt.liquid file contains the following core Liquid loop:

# robots.txt.liquid
{% for group in robots.default_groups %}
  {{- group.user_agent -}}

  {% for rule in group.rules %}
    {{- rule -}}
  {% endfor %}

  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

This loop pulls Shopify's platform-managed disallow rules, protecting critical checkout infrastructure:

• Protected Checkout Endpoints: /cart, /orders, /checkouts/, and /checkout are automatically disallowed to prevent search bots from indexing abandoned carts and private user checkouts.
• Internal Search & Policies: /search and /policies/ are excluded to prevent crawl budget waste on thin search parameter URLs.
• Faceted Collection Filters: Shopify by default adds directives preventing indexation of multi-tag collection permutations (e.g., /collections/*/* or URL parameters containing filter.).

How to Safely Append Custom Rules in Shopify

When customizing your Shopify robots directives, never delete the for group in robots.default_groups loop entirely. Instead, inject your custom rules directly below or within conditional blocks:

# Appending Custom AI Restrictions in Shopify robots.txt.liquid
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Bytespider
Disallow: /

# Retain Shopify Platform Core Rules
{% for group in robots.default_groups %}
  {{- group.user_agent -}}
  {% for rule in group.rules %}
    {{- rule -}}
  {% endfor %}
  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}