FREEFirst project with us? Your landing page design is on the house.See the offer

Free tool

AI crawler
checker.

See which AI and search bots your site lets in, and get a robots.txt that says what you actually want.

This tool reads your robots.txt and works out whether 21 AI and search bots may fetch a page, using longest match rules. It groups them by training, search and user requests, and checks meta robots and X-Robots-Tag. robots.txt is a request, not a lock, and it cannot see firewall or CDN blocks.

We fetch robots.txt and the page using the name KookiesAEOCheck. We do not store the URL or the results.

Or paste a robots.txt

Runs only in your browser. Nothing you paste is sent anywhere.

Who we check

21 bots,
from vendor docs.

We checked vendor docs on 7 November 2026. Two exceptions: we found no page from ByteDance for Bytespider, and we could not read Microsoft's page for Bingbot that day. New bots appear often, so tell us if one is missing.

Training

Bots that collect content that may train AI models. Blocking these does not take you out of AI search.

GPTBotOpenAI

Collects content that may train OpenAI models.

ClaudeBotAnthropic

Collects content that may train Anthropic models.

Google-ExtendedGoogle

A control token, not a crawler. Says if Google may use crawled pages for Gemini training and grounding. It does not affect Google Search.

Applebot-ExtendedApple

A control token, not a crawler. Says if Apple may use pages crawled by Applebot to train its AI models.

Meta-ExternalAgentMeta

Indexes content to train AI models or improve products.

AmazonbotAmazon

Improves Amazon products and answers, and may train Amazon AI models.

CCBotCommon Crawl

Builds an open web dataset that anyone can use, which includes AI teams.

BytespiderByteDance

Widely reported as a crawler for ByteDance AI. We found no ByteDance page to confirm this.

Search

Bots that build search indexes, including AI search.

GooglebotGoogle

The main Google Search crawler. Blocking it takes you out of Search, and with it the AI answers there.

BingbotMicrosoft

The main Bing crawler.

ApplebotApple

Powers search in Spotlight, Siri and Safari. Apple says its data may also help train AI models, unless Applebot-Extended is blocked.

OAI-SearchBotOpenAI

Shows sites in ChatGPT search answers. Not for training.

Claude-SearchBotAnthropic

Improves search results in Claude.

PerplexityBotPerplexity

Lists and links sites in Perplexity results. Not for training.

Meta-WebIndexerMeta

Helps Meta AI search find and cite content.

Amzn-SearchBotAmazon

Powers search at Amazon, including Alexa. Not for training.

User requests

Bots that visit a page when a person asks. Some vendors say these may not follow robots.txt.

ChatGPT-UserOpenAI

Visits a page when a ChatGPT user asks. OpenAI says robots.txt rules may not apply.

Claude-UserAnthropic

Visits a page when a Claude user asks.

Perplexity-UserPerplexity

Visits a page when a Perplexity user asks. Perplexity says it generally ignores robots.txt.

Meta-ExternalFetcherMeta

Fetches a link when a Meta AI user asks. Meta says it may bypass robots.txt.

Amzn-UserAmazon

Fetches pages when an Alexa user asks. Amazon says it may not follow all robots.txt rules.

FAQs

AI crawler
questions.

Abdul Quadir
Still have a question?Ask us directly. We reply within a day.
Book a callPick a timehello@kookies.io

That is a business call. Blocking training bots such as GPTBot or ClaudeBot asks those vendors not to use your future content for training, and they are separate from their search bots. Many marketing sites want to be read and allow them. Publishers often block them.

Not by itself. OpenAI says GPTBot is for training, OAI-SearchBot is for ChatGPT search, and ChatGPT-User is for visits a user asks for. Each has its own rule.

We only read robots.txt and the page. Firewalls, CDN bot rules and hosts can block a bot by name or address, and we cannot see that from here. Check your host or CDN settings too.

A control token, not a separate crawler. Google says it controls use of content for Gemini training and grounding, and does not affect Google Search.

It follows the robots.txt standard, RFC 9309. A bot uses the group that names it, or the * group if none does. The longest matching rule wins, and allow wins a tie.

No. Pasted text never leaves your browser. For a URL we fetch the files, work out the result and send it to you, and we do not save the URL or the results.

Your turn

Let's kook
what you krave.

Hi Kookies, I need a built on . I'd love to launch . Reach me at
or book a callSent. We reply within one working day.