Free tool
AI crawler
checker.
See which AI and search bots your site lets in, and get a robots.txt that says what you actually want.
This tool reads your robots.txt and works out whether 21 AI and search bots may fetch a page, using longest match rules. It groups them by training, search and user requests, and checks meta robots and X-Robots-Tag. robots.txt is a request, not a lock, and it cannot see firewall or CDN blocks.
We fetch robots.txt and the page using the name KookiesAEOCheck. We do not store the URL or the results.
Reading robots.txt and the page...
Or paste a robots.txt
Runs only in your browser. Nothing you paste is sent anywhere.
robots.txt is a request, not a lock: polite bots follow it and others may not. Firewall and CDN blocks do not show up here, so a bot we list as allowed may still be blocked. Your site's own settings decide that.
Suggested robots.txt
Your other rules and sitemaps are kept. Read it before you publish, because this is your call.
Who we check
21 bots,
from vendor docs.
We checked vendor docs on 7 November 2026. Two exceptions: we found no page from ByteDance for Bytespider, and we could not read Microsoft's page for Bingbot that day. New bots appear often, so tell us if one is missing.
Training
Bots that collect content that may train AI models. Blocking these does not take you out of AI search.
GPTBotOpenAICollects content that may train OpenAI models.
ClaudeBotAnthropicCollects content that may train Anthropic models.
Google-ExtendedGoogleA control token, not a crawler. Says if Google may use crawled pages for Gemini training and grounding. It does not affect Google Search.
Applebot-ExtendedAppleA control token, not a crawler. Says if Apple may use pages crawled by Applebot to train its AI models.
Meta-ExternalAgentMetaIndexes content to train AI models or improve products.
AmazonbotAmazonImproves Amazon products and answers, and may train Amazon AI models.
CCBotCommon CrawlBuilds an open web dataset that anyone can use, which includes AI teams.
BytespiderByteDanceWidely reported as a crawler for ByteDance AI. We found no ByteDance page to confirm this.
Search
Bots that build search indexes, including AI search.
GooglebotGoogleThe main Google Search crawler. Blocking it takes you out of Search, and with it the AI answers there.
BingbotMicrosoftThe main Bing crawler.
ApplebotApplePowers search in Spotlight, Siri and Safari. Apple says its data may also help train AI models, unless Applebot-Extended is blocked.
OAI-SearchBotOpenAIShows sites in ChatGPT search answers. Not for training.
Claude-SearchBotAnthropicImproves search results in Claude.
PerplexityBotPerplexityLists and links sites in Perplexity results. Not for training.
Meta-WebIndexerMetaHelps Meta AI search find and cite content.
Amzn-SearchBotAmazonPowers search at Amazon, including Alexa. Not for training.
User requests
Bots that visit a page when a person asks. Some vendors say these may not follow robots.txt.
ChatGPT-UserOpenAIVisits a page when a ChatGPT user asks. OpenAI says robots.txt rules may not apply.
Claude-UserAnthropicVisits a page when a Claude user asks.
Perplexity-UserPerplexityVisits a page when a Perplexity user asks. Perplexity says it generally ignores robots.txt.
Meta-ExternalFetcherMetaFetches a link when a Meta AI user asks. Meta says it may bypass robots.txt.
Amzn-UserAmazonFetches pages when an Alexa user asks. Amazon says it may not follow all robots.txt rules.
FAQs
AI crawler
questions.

That is a business call. Blocking training bots such as GPTBot or ClaudeBot asks those vendors not to use your future content for training, and they are separate from their search bots. Many marketing sites want to be read and allow them. Publishers often block them.
Not by itself. OpenAI says GPTBot is for training, OAI-SearchBot is for ChatGPT search, and ChatGPT-User is for visits a user asks for. Each has its own rule.
We only read robots.txt and the page. Firewalls, CDN bot rules and hosts can block a bot by name or address, and we cannot see that from here. Check your host or CDN settings too.
A control token, not a separate crawler. Google says it controls use of content for Gemini training and grounding, and does not affect Google Search.
It follows the robots.txt standard, RFC 9309. A bot uses the group that names it, or the * group if none does. The longest matching rule wins, and allow wins a tie.
No. Pasted text never leaves your browser. For a URL we fetch the files, work out the result and send it to you, and we do not save the URL or the results.


