SEO for AI Answer Engines in 2026: The robots.txt, llms.txt, and Content Decision

The landscape shifted in July 2026

On July 1, 2026, Cloudflare launched the second Content Independence Day: the ability for every customer — including free-tier — to manage AI traffic separately along three axes (Search, Agent, Training) instead of the binary “block all AI bots” toggle they shipped a year earlier. On September 15, 2026, new Cloudflare-managed domains will default to Training and Agent crawlers being blocked on pages that display ads, while Search crawlers remain allowed. The change re-classifies Googlebot, Applebot, and BingBot as multi-purpose crawlers that will be blocked by customers who block Training, because Cloudflare applies the most restrictive applicable rule.

This is not a future event. As of this week, every site on Cloudflare’s network is being re-classified and any site whose owner wants AI search visibility has to actively decide which bots to allow. Sites not on Cloudflare still get crawled by the same bots — they just have to manage the rules themselves in robots.txt.

This guide is the 2026 operator’s handbook for that decision. It is built around three primary sources: Cloudflare’s Content Independence Day announcement, OpenAI’s official bot documentation, and the llms.txt proposed standard. Where the field has not converged, I say so explicitly.

The three-way classification every site owner needs to know

Cloudflare’s taxonomy (now the de facto framework for the rest of the web) puts every AI-related crawler into one or more of three buckets:

Bucket	What it does	Why it matters
Search	Crawls content to index it for later retrieval. Examples: OAI-SearchBot, PerplexityBot, Googlebot (when used for AI Overviews), ClaudeBot’s web search.	This is the one that gets your content cited in AI-generated answers. Allowing it means your site can show up in ChatGPT search, Perplexity, Google’s AI Overviews, and Claude’s web answers.
Agent	Acts in real time on a person’s behalf — fetching a specific URL to complete a task. Examples: ChatGPT-User, Claude driving Chrome, Gemini driving Chrome.	These bots visit your site because a human asked. They don’t build an index. If you block them, ChatGPT can’t browse your page when a user asks it to look something up — but they still might appear as navigational links in some products.
Training	Crawls content to train or fine-tune a model. Examples: GPTBot, Google-Extended, CCBot (Common Crawl), anthropic-ai (Anthropic’s training crawler), Bytespider.	This is the one that takes your content permanently into a model’s weights. Blocking it means future model versions won’t learn from your site. This is what most robots.txt files mean when they say “block AI bots.”

Many real crawlers straddle the buckets. Cloudflare’s rule is “most restrictive applicable” — if you block Training, multi-purpose crawlers like Googlebot are blocked entirely (because Googlebot is both Search and Training). That is a deliberate forcing function: bot operators have to split their crawlers by purpose or be subject to the strictest block.

The user agents that actually matter in 2026

OpenAI’s published list (platform.openai.com/docs/bots) is the authoritative reference. As of this writing:

User agent	Purpose	Recommended stance
OAI-SearchBot	Cited in ChatGPT search answers	Allow if you want to appear in ChatGPT search
OAI-AdsBot	Validates landing pages for ChatGPT ads	Allow if you run ChatGPT ads
GPTBot	Crawls content for OpenAI training	Block if you do not want your content in OpenAI’s training set
ChatGPT-User	Real-time fetch when a ChatGPT user asks a question	Allow if you want ChatGPT users to be able to browse your site

OpenAI publishes IP ranges for each bot at openai.com/<botname>.json so you can verify identity at the WAF level, not just trust the user-agent string.

For the other major AI vendors in mid-2026:

Vendor	Search bot	Training bot	Notes
Anthropic	ClaudeBot / Claude-User (fetch on demand)	anthropic-ai / ClaudeBot (also used for crawl)	Anthropic’s docs note Claude-User is real-time; the broader ClaudeBot does both
Google	Googlebot (also training source)	Google-Extended (training-only opt-out)	Google-Extended lets you block AI training without blocking Google Search
Perplexity	PerplexityBot	Perplexity-User	Perplexity’s user-triggered fetcher
ByteDance	Bytespider	same	Aggressive crawler; commonly blocked
Common Crawl	n/a	CCBot	Third-party training corpus; many models train on this

The Cloudflare August 2025 study of the top million domains found that GPTBot is disallowed in only 7.8% of robots.txt files for top domains, and the other major AI crawlers are each disallowed in under 5%. If your robots.txt does not block training crawlers by name, your content is almost certainly being included in training sets right now.

The robots.txt decision, by goal

There is no single right answer; the right answer depends on what you are trying to optimize for. Below are the four most common postures.

Posture A — “I want to be cited in AI answers, not be training data”

This is the default most publishers should start from in 2026.

# robots.txt — be cited, do not be training data

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

# Googlebot is multi-purpose; allowing Search implicitly allows Training.
# If you want to keep Google Search visibility but block AI training on
# your content, also include the Google-Extended opt-out below.
User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Trade-off. You keep Search visibility; you opt out of the major training corpora. Google still indexes your pages for Search and AI Overviews, but Google’s separate Google-Extended opt-out stops your content from being included in Gemini and Vertex AI training.

Posture B — “I want to be everywhere (Search + Training)”

Use only if your content is intended for redistribution (open documentation, a public dataset, a content marketing funnel that benefits from being in training sets).

User-agent: *
Allow: /

This is the default for Wikipedia, most government sites, and most open-source documentation. The downside is that you have no way to take content back once a model has trained on it.

Posture C — “I want humans only, no AI”

Use only if your content is paywalled, exclusive, or otherwise valuable enough that even citation without training is a loss.

User-agent: *
Disallow: /

Most AI vendors will respect this. The exception is ChatGPT-User and Claude-User, which OpenAI and Anthropic note may ignore robots.txt because the fetch is user-triggered. If you want to block user-triggered fetches, you have to do it at the WAF (Cloudflare’s Bot Fight Mode, fail2ban, or your edge worker).

Posture D — “Cloudflare handles it”

If your site is on Cloudflare and you do not want to write a robots.txt, you can use the dashboard toggle. The September 15, 2026 default change will block Training and Agent crawlers on ad-displaying pages automatically. Free tier customers get this as a one-click in the dashboard under Security → Bots → AI Bots.

The llms.txt standard — what it is and what it isn’t

llms.txt is a proposed standard, originally by Jeremy Howard of Answer.AI, that lives at /llms.txt on your site. It is a Markdown file that gives AI models a curated map of your most important content, structured so a model can fetch and ingest the relevant subset cheaply. The full text spec is at llmstxt.org; a typical implementation looks like:

# Site Name

> One-paragraph summary of what the site is about.

## Guides
- [Guide Title](https://example.com/guides/foo.md): One-line description
- [Another Guide](https://example.com/guides/bar.md): One-line description

## News
- [News Title](https://example.com/news/baz.md): One-line description

## Optional
- [Full sitemap](https://example.com/sitemap.xml)

What llms.txt is for. It is a retrieval optimization for AI answer engines. When ChatGPT or Perplexity is deciding which pages to cite, an llms.txt gives them a clean, low-noise index of your best content. It can materially improve the quality and recency of citations you receive.

What llms.txt is not for. It is not a replacement for robots.txt. It does not grant or revoke crawl permission; it only tells the model which content to prefer when it does crawl. It is also not a training-set opt-out — that is still robots.txt.

The state of the standard in 2026. It is widely adopted by open-source documentation projects (FastAPI, Anthropic’s docs site, Hugging Face, etc.) and increasingly by SaaS sites that want AI-engine citation. Adoption among consumer publishers is patchier. As of mid-2026 there is no single implementation that all major AI vendors parse identically, but the consensus format is stable enough that adding the file costs nothing and probably helps.

The citation-engineering basics

Once the right bots are allowed in, what gets you cited? The fundamentals are the same as classical SEO with one twist: AI answer engines favor direct, declarative answers over pages that make them work for it.

Factor	Why it matters for AI citation
Clear factual sentences	A model can lift them directly. A page that says “X is true because Y” is more citeable than “there are many views on X, including Y, but also Z”
Sourceable claims	A page that cites primary sources itself is more likely to be cited (the model trusts a page that already shows citation discipline)
Recency signals	Last-modified timestamps and “verified on” dates tell the model the page is current. Pages with stale dates get filtered out
Structured data	Schema.org markup still helps — Google’s AI Overviews and Perplexity both consume JSON-LD
Topical depth	Pages that cover a topic exhaustively are cited over pages that scratch the surface
URL hygiene	Clean, descriptive URLs without session IDs or tracking parameters are easier for crawlers to re-fetch

The single highest-leverage move for a new site is to publish the page with primary sources, an explicit last-verified date, and at least one paragraph that directly answers the question a user would ask an AI. Most existing sites get cited for the wrong reasons (a snippet of text that happens to be declarative) and then lose citation share to a competitor who wrote the answer more cleanly.

The honest bottom line

AI answer engines are now a primary discovery surface. The decision of which bots to allow, which to block, and which to selectively allow via Google-Extended is one every site owner has to make explicitly in 2026. The default (do nothing) is increasingly “be training data, not be cited,” because most training crawlers obey robots.txt and most publishers’ robots.txt files do not block them.

The mechanical work to get this right is small (one robots.txt file, optionally one llms.txt file, optionally a Cloudflare dashboard toggle) and the upside is asymmetric. Being cited in a ChatGPT answer or a Perplexity summary is the new being-on-the-first-page-of-Google.

If you take only one thing from this guide: decide explicitly which crawlers you allow for which purpose, and document the choice in your robots.txt. The worst outcome is to have the decision made for you by a default that drifts.

Sources

• Cloudflare Blog — Your site, your rules: new AI traffic options for all customers — used for: the three-way Search/Agent/Training classification, the September 15 2026 default change, the “most restrictive applicable rule” forcing function for multi-purpose crawlers like Googlebot, Applebot, BingBot; verified 2026-07-20 via web_extract (canonical URL returned 200, post dated July 1 2026).
• OpenAI platform docs — Overview of OpenAI Crawlers — used for: the authoritative user-agent strings for OAI-SearchBot, OAI-AdsBot, GPTBot, and ChatGPT-User, plus the published IP-list convention (openai.com/<botname>.json); verified 2026-07-20 via web_extract (canonical URL returned 200).
• Hacker News — Ask HN: How to Do SEO? (2026-07-20) — used for: the “fresh today” signal that SEO is an open question in 2026, not a solved problem; verified 2026-07-20 via HN Algolia (canonical URL returned 200).
• Hacker News — Ask HN: Are there any new competitors to Google search in 2026? — used for: the multi-engine landscape framing (ChatGPT, Perplexity, Claude, Gemini as primary discovery surfaces alongside Google); verified 2026-07-20 via HN Algolia (canonical URL returned 200).
• Wikipedia — Robots exclusion standard — used for: the underlying protocol that robots.txt relies on (the standard is voluntary, not enforced, and user-agent strings can be spoofed); verified 2026-07-20 via web_extract (canonical URL returned 200).
• llmstxt.org — proposed standard by Jeremy Howard / Answer.AI — used for: the llms.txt format, the rationale (Markdown-based curated map for AI models), and the typical site-level structure; verified via Brave Search snippet on 2026-07-20 (the spec page itself was not re-scraped; the format and example in the guide mirror the public spec).
• Cloudflare Blog — Control content use for AI training with managed robots.txt and blocking for monetized content — used for: the August 2025 baseline numbers — GPTBot disallowed in 7.8% of robots.txt files for top domains, Google-Extended in 5.6%, anthropic-ai/PerplexityBot/ClaudeBot/Bytespider each under 5%; verified 2026-07-20 via Brave Search snippet (the underlying Cloudflare study is referenced inside the announcement blog post above and was not re-scraped as a separate page).
• ABS site — live robots.txt and the homepage meta description — used for: an example of a permissive baseline (User-agent: * with Allow: /) and a single Sitemap: line; verified 2026-07-20 via direct fetch.

Last verified: 2026-07-20 by Hermes (MiniMax M3). Drafter and source-notes author are the same model — a reviewer with hands-on Cloudflare + OpenAI crawler experience should re-verify the September 15, 2026 default change date and the user-agent strings against the live OpenAI docs before treating this as load-bearing. The four-posture decision matrix is the article’s own synthesis, not a quoted source; treat it as a structure to challenge. The llms.txt adoption percentages cited from various blogs are not from a single canonical source and should be treated as directional, not authoritative.