FREEFirst project with us? Your landing page design is on the house.See the offer

Glossary · Technical SEO

What is robots.txt?

By Abdul QuadirUpdated Nov 2026
robots.txtnoun · Technical SEO

robots.txt is a small text file at the root of your site that tells crawlers which parts they may visit. It also points them to your sitemap.

robots.txt is a small text file at the root of your site that tells crawlers which parts they may visit. It also points them to your sitemap.

robots.txt, explained

Crawlers from search engines and AI tools check robots.txt before they read a site. Each rule names a crawler (a user agent) and the paths it may or may not visit.

It is used to keep crawlers out of things like admin areas, search result pages or staging folders, and to save crawl budget on large sites.

robots.txt controls crawling, not indexing. A blocked page can still show in search if other sites link to it. To keep a page out of results, use a noindex tag.

Today it is also where you decide which AI crawlers, like GPTBot or ClaudeBot, may read your content.

In our work

How Kookies uses it

We set robots.txt so search engines and AI crawlers can read your public pages, while private and utility pages stay out. On this site, AI crawlers are allowed, because we want to be cited.

FAQs

robots.txt
questions.

Abdul Quadir
Still have a question?Ask us directly. We reply within a day.
Book a callPick a timehello@kookies.io

At the root of your domain, for example www.example.com/robots.txt.

If you want to appear in AI answers, no. Blocking them means tools like ChatGPT and Claude cannot read or cite your pages.

Yes, in Site settings under SEO.

Your turn

Let's kook
what you krave.

Hi Kookies, I need a built on . I'd love to launch . Reach me at
or book a callSent. We reply within one working day.