FREEFirst project with us? Your landing page design is on the house.See the offer

Glossary · AI search

What is AI crawler?

By Abdul QuadirUpdated Nov 2026
AI crawlernoun · AI search

An AI crawler is a bot that visits web pages for AI companies. Some collect content for model training, some build a search index, and some fetch a page when a user asks.

An AI crawler is a bot that visits web pages for AI companies. Some collect content for model training, some build a search index, and some fetch a page when a user asks.

AI crawler, explained

Those jobs are often separate. OpenAI, for example, uses different agents for training and for search, so you can allow one and block another.

You set preferences in robots.txt by naming each user agent. Reputable bots say they respect it, but robots.txt is a request, not a security wall, and user-triggered fetches can be treated differently.

Think about the trade-off before blocking. Blocking a search crawler can mean you do not appear as a source in that product, whereas blocking a training crawler has a different effect.

Check your server logs or your CDN to see which bots visit. Also look at firewall settings, since a CDN rule can block a bot you meant to allow.

In our work

How Kookies uses it

Our own robots.txt allows the main AI crawlers, and we set up crawler rules with clients based on what they want, as part of SEO, AEO and GEO work.

FAQs

AI crawler
questions.

Abdul Quadir
Still have a question?Ask us directly. We reply within a day.
Book a callPick a timehello@kookies.io

It depends on your goals. If you want to appear as a source in AI answers, keep the search crawlers open. Publishers with paid content may choose differently.

Blocking Google-Extended does not affect Google Search, according to Google. Blocking Googlebot would.

Do not rely on it. Putting key content in the HTML is the safer choice.

Your turn

Let's kook
what you krave.

Hi Kookies, I need a built on . I'd love to launch . Reach me at
or book a callSent. We reply within one working day.