AI crawler, explained
Those jobs are often separate. OpenAI, for example, uses different agents for training and for search, so you can allow one and block another.
You set preferences in robots.txt by naming each user agent. Reputable bots say they respect it, but robots.txt is a request, not a security wall, and user-triggered fetches can be treated differently.
Think about the trade-off before blocking. Blocking a search crawler can mean you do not appear as a source in that product, whereas blocking a training crawler has a different effect.
Check your server logs or your CDN to see which bots visit. Also look at firewall settings, since a CDN rule can block a bot you meant to allow.



