robots.txt, explained
Crawlers from search engines and AI tools check robots.txt before they read a site. Each rule names a crawler (a user agent) and the paths it may or may not visit.
It is used to keep crawlers out of things like admin areas, search result pages or staging folders, and to save crawl budget on large sites.
robots.txt controls crawling, not indexing. A blocked page can still show in search if other sites link to it. To keep a page out of results, use a noindex tag.
Today it is also where you decide which AI crawlers, like GPTBot or ClaudeBot, may read your content.



