FREEFirst project with us? Your landing page design is on the house.See the offer

Chapter 4 of 6

llms.txt and AI crawlers

Who crawls for what, your robots.txt policy, firewall checks and when llms.txt is worth it.

5 min readUpdated 2026-11-07
All chaptersChapter 4 of 6
  1. 01What AEO is and what changes
  2. 02Answer-first copy
  3. 03Schema per page type
  4. 04llms.txt and AI crawlers
  5. 05FAQ, comparison and glossary pages
  6. 06Measuring it, and what nobody can promise

The short answer

Crawler access comes first and llms.txt comes a long way second. Decide which AI crawlers may visit, set that in robots.txt, then confirm your host or firewall is not blocking them anyway. After that, an llms.txt file is a cheap, optional extra. Google says you do not need AI text files or new machine-readable files for its AI features. The llms.txt site itself describes the format as a proposal.

Who is crawling, and for what

This table comes from each vendor's own crawler page. Names and behaviour change, so check the current docs before you act.

NameCompanyJobWhat blocking does
GPTBotOpenAIContent that may be used to train modelsSignals the content should not be used in training
OAI-SearchBotOpenAIHelps sites appear in ChatGPT searchNot shown in ChatGPT search answers, though navigational links can still appear
ChatGPT-UserOpenAIActions started by a userOpenAI says robots.txt rules may not apply
ClaudeBotAnthropicContent that may be used to train modelsFuture material should be excluded from training
Claude-SearchBotAnthropicImproves search resultsMay reduce visibility in search responses
Claude-UserAnthropicFetches pages for a user's questionClaude cannot retrieve the site for that question
PerplexityBotPerplexitySurfaces and links sites in its results, not for training foundation modelsPerplexity recommends allowing it
Perplexity-UserPerplexityVisits a page when a user asksPerplexity says it generally ignores robots.txt
Google-ExtendedGoogleA control token for Gemini training and grounding, not a separate crawlerDoes not affect inclusion or ranking in Google Search

Two details are easy to miss. OpenAI says each of its settings is independent, so you can allow search and refuse training. And Google-Extended is not the control for what Search shows. To limit that, Google points to nosnippet, data-nosnippet, max-snippet or noindex.

Choosing your robots.txt policy

There are three sensible positions. Pick one on purpose.

PolicySuitsSet in robots.txt
Allow allMarketing sites that want reachNo blocks. Naming each crawler is optional, because User-agent: * with Allow: / covers those without their own rule.
Search yes, training noTeams that want to appear in answers but keep content out of training where they canAdd Disallow: / under User-agent: GPTBot, User-agent: ClaudeBot and User-agent: Google-Extended.
Block mostPaid content, licensed data, members' materialBlock search crawlers too, and put real access control on the content. Robots rules are a request.

This is a business choice, and for licensed content a legal one. We are not giving legal advice here, so ask your lawyer if your rights to the content are in question.

Some practical notes. Anthropic's rules apply per subdomain, so repeat them for each one. Anthropic supports the non-standard Crawl-delay if a crawler is loading your server. OpenAI says robots.txt changes can take about a day to affect search. Our own robots.txt uses the first policy.

robots.txt is not the only gate

A site can say yes in robots.txt while its firewall says no. Google's guidance names the CDN and hosting layer as places to allow crawling, and it is easy to miss because it never appears in your code.

  • Check your firewall and bot settings. Look at your CDN's bot rules and security events for blocked AI user agents.
  • Match the name and the address. Anyone can fake a user agent. OpenAI, Anthropic and Perplexity all publish IP lists, so allow rules should use both. Perplexity's docs give Cloudflare and AWS WAF examples.
  • Know the Cloudflare catch. Perplexity notes that Bot Fight Mode cannot be skipped with WAF custom rules and may still challenge its bot.
  • Do not block by IP to opt out. Anthropic warns against it, because a blocked bot cannot read your robots.txt, so the opt-out may not work reliably.
  • Keep lists fresh. Perplexity says its ranges change and suggests automating updates.

Test what your site tells each crawler with the AI crawler checker. Read about the wider group in our AI crawler entry.

llms.txt: when it is worth it

The format, from llmstxt.org, is short: a Markdown file at /llms.txt with an H1 name, a blockquote summary and H2 sections of links with notes. An H2 called Optional marks links an agent can skip. The site describes the file as read on demand when an agent needs context for a user's request, which is a different job from robots.txt or a sitemap.

Our llms.txt guide has a full example, so here is the decision rule only.

  • Add it if you can keep it in sync with little effort, ideally from your build or your CMS.
  • Skip it for now if access, copy or schema are unfinished. They come first.
  • Prioritise it if you have documentation or an API. A curated index read on demand fits that kind of content well, in our view.
  • Do not expect a Google benefit. Google has said it is not needed for its AI features, and we have not read a vendor page that promises one.

Make a first version with the llms.txt generator. Then write the summary yourself. It should state what you sell, to whom, and your key facts in plain sentences.

How we generate ours

A hand-kept file goes stale. Ours does not, because a script runs after each build. It walks the built pages, skips anything marked noindex, reads each title and description, and writes grouped links: company pages first, then services, migrations, comparisons, industries, case studies, blog, locations and glossary. The summary at the top holds the prices and contact details. It also writes llms-full.txt with the full text of core pages, and a separate llms-integrations.txt so the main file is not swamped.

The spec also suggests Markdown copies of pages at matching URLs. Our build writes the full-text files instead. Pick whichever you can maintain.

Access checklist

  • You have written down your policy: allow all, search yes and training no, or block most.
  • robots.txt matches that policy and returns a 200 on every subdomain you care about.
  • A sitemap is linked from robots.txt.
  • CDN or host bot rules do not contradict robots.txt.
  • Staging rules, such as a site-wide Disallow or noindex, did not reach production.
  • Key pages are indexed and allowed a snippet. Check with URL Inspection.
  • Important text appears in the HTML source.
  • If you have an llms.txt, it opens, its links work and its prices match the site.
  • Someone owns a review of this list after any platform or host change.

Takeaways

What to remember from chapter 4.

  • Crawler access comes first. llms.txt is a cheap, optional extra.
  • You can allow AI search and refuse training, because the vendors separate them.
  • Check your CDN and firewall, because robots.txt is not the only gate.
  • Google-Extended does not affect inclusion or ranking in Google Search.

Your turn

Let's kook
what you krave.

Hi Kookies, I need a built on . I'd love to launch . Reach me at
or book a callSent. We reply within one working day.