The short answer
Crawler access comes first and llms.txt comes a long way second. Decide which AI crawlers may visit, set that in robots.txt, then confirm your host or firewall is not blocking them anyway. After that, an llms.txt file is a cheap, optional extra. Google says you do not need AI text files or new machine-readable files for its AI features. The llms.txt site itself describes the format as a proposal.
Who is crawling, and for what
This table comes from each vendor's own crawler page. Names and behaviour change, so check the current docs before you act.
| Name | Company | Job | What blocking does |
|---|---|---|---|
| GPTBot | OpenAI | Content that may be used to train models | Signals the content should not be used in training |
| OAI-SearchBot | OpenAI | Helps sites appear in ChatGPT search | Not shown in ChatGPT search answers, though navigational links can still appear |
| ChatGPT-User | OpenAI | Actions started by a user | OpenAI says robots.txt rules may not apply |
| ClaudeBot | Anthropic | Content that may be used to train models | Future material should be excluded from training |
| Claude-SearchBot | Anthropic | Improves search results | May reduce visibility in search responses |
| Claude-User | Anthropic | Fetches pages for a user's question | Claude cannot retrieve the site for that question |
| PerplexityBot | Perplexity | Surfaces and links sites in its results, not for training foundation models | Perplexity recommends allowing it |
| Perplexity-User | Perplexity | Visits a page when a user asks | Perplexity says it generally ignores robots.txt |
| Google-Extended | A control token for Gemini training and grounding, not a separate crawler | Does not affect inclusion or ranking in Google Search |
Two details are easy to miss. OpenAI says each of its settings is independent, so you can allow search and refuse training. And Google-Extended is not the control for what Search shows. To limit that, Google points to nosnippet, data-nosnippet, max-snippet or noindex.
Choosing your robots.txt policy
There are three sensible positions. Pick one on purpose.
| Policy | Suits | Set in robots.txt |
|---|---|---|
| Allow all | Marketing sites that want reach | No blocks. Naming each crawler is optional, because User-agent: * with Allow: / covers those without their own rule. |
| Search yes, training no | Teams that want to appear in answers but keep content out of training where they can | Add Disallow: / under User-agent: GPTBot, User-agent: ClaudeBot and User-agent: Google-Extended. |
| Block most | Paid content, licensed data, members' material | Block search crawlers too, and put real access control on the content. Robots rules are a request. |
This is a business choice, and for licensed content a legal one. We are not giving legal advice here, so ask your lawyer if your rights to the content are in question.
Some practical notes. Anthropic's rules apply per subdomain, so repeat them for each one. Anthropic supports the non-standard Crawl-delay if a crawler is loading your server. OpenAI says robots.txt changes can take about a day to affect search. Our own robots.txt uses the first policy.
robots.txt is not the only gate
A site can say yes in robots.txt while its firewall says no. Google's guidance names the CDN and hosting layer as places to allow crawling, and it is easy to miss because it never appears in your code.
- Check your firewall and bot settings. Look at your CDN's bot rules and security events for blocked AI user agents.
- Match the name and the address. Anyone can fake a user agent. OpenAI, Anthropic and Perplexity all publish IP lists, so allow rules should use both. Perplexity's docs give Cloudflare and AWS WAF examples.
- Know the Cloudflare catch. Perplexity notes that Bot Fight Mode cannot be skipped with WAF custom rules and may still challenge its bot.
- Do not block by IP to opt out. Anthropic warns against it, because a blocked bot cannot read your robots.txt, so the opt-out may not work reliably.
- Keep lists fresh. Perplexity says its ranges change and suggests automating updates.
Test what your site tells each crawler with the AI crawler checker. Read about the wider group in our AI crawler entry.
llms.txt: when it is worth it
The format, from llmstxt.org, is short: a Markdown file at /llms.txt with an H1 name, a blockquote summary and H2 sections of links with notes. An H2 called Optional marks links an agent can skip. The site describes the file as read on demand when an agent needs context for a user's request, which is a different job from robots.txt or a sitemap.
Our llms.txt guide has a full example, so here is the decision rule only.
- Add it if you can keep it in sync with little effort, ideally from your build or your CMS.
- Skip it for now if access, copy or schema are unfinished. They come first.
- Prioritise it if you have documentation or an API. A curated index read on demand fits that kind of content well, in our view.
- Do not expect a Google benefit. Google has said it is not needed for its AI features, and we have not read a vendor page that promises one.
Make a first version with the llms.txt generator. Then write the summary yourself. It should state what you sell, to whom, and your key facts in plain sentences.
How we generate ours
A hand-kept file goes stale. Ours does not, because a script runs after each build. It walks the built pages, skips anything marked noindex, reads each title and description, and writes grouped links: company pages first, then services, migrations, comparisons, industries, case studies, blog, locations and glossary. The summary at the top holds the prices and contact details. It also writes llms-full.txt with the full text of core pages, and a separate llms-integrations.txt so the main file is not swamped.
The spec also suggests Markdown copies of pages at matching URLs. Our build writes the full-text files instead. Pick whichever you can maintain.
Access checklist
- You have written down your policy: allow all, search yes and training no, or block most.
- robots.txt matches that policy and returns a 200 on every subdomain you care about.
- A sitemap is linked from robots.txt.
- CDN or host bot rules do not contradict robots.txt.
- Staging rules, such as a site-wide
Disallowornoindex, did not reach production. - Key pages are indexed and allowed a snippet. Check with URL Inspection.
- Important text appears in the HTML source.
- If you have an llms.txt, it opens, its links work and its prices match the site.
- Someone owns a review of this list after any platform or host change.


