The Comprehensive Guide to Meta Robots Directives & Google Discover
Comprehensive Technical Guide & Best Practices
1How Search Engines Interpret Robots Meta Tags
The robots meta tag (<meta name="robots" content="...">) gives website owners granular, page-by-page control over how search crawlers index content and pass link equity. While robots.txt blocks crawlers from accessing URLs entirely, the robots meta tag allows crawlers to fetch the page while strictly obeying indexing restrictions (such as indexing without following links, or vice-versa).
Crucially, modern search engines like Google also support fine-grained preview controls (such as max-image-preview:large and max-snippet:-1) which directly determine your eligibility for Google Discover traffic.
- noindex stops a page from appearing in search results while allowing search engines to crawl it.
- max-image-preview:large is mandatory for maximizing Google Discover CTR and impression volume.
- Do not block noindexed pages in robots.txt; doing so prevents crawlers from reading the noindex tag.
2Meta Robots Directives Breakdown
Key directives supported across major search engines in 2026:
- index / noindex: Tells search engines whether to store the page in their search index.
- follow / nofollow: Tells crawlers whether to follow links found on this page to discover other URLs.
- max-image-preview:large: Authorizes Google to display full-width high-resolution images in search cards and Discover.
- max-snippet:[number]: Limits the text snippet length in search results.
- noarchive: Prevents search engines from caching HTML snapshots of the page.
- Combine index, follow, max-image-preview:large as the gold standard for marketing pages.
- Use noindex, follow on paginated blog category archives to pass equity to older articles without indexing thin lists.