Research · AI crawlers
Do top cloud companies block AI crawlers?
We read the robots.txt file of every company on the Forbes Cloud 100 (2025) list and checked the rules for 15 AI crawlers and fetchers.
Almost none of them block AI crawlers. Of the 94 companies whose robots.txt we could read, 94% let every AI crawler we checked read their home page. Only 1 blocks OpenAI's GPTBot and 1 blocks Anthropic's ClaudeBot. Most never mention AI crawlers at all: 21% name one in robots.txt.
Findings
What we found.
4 things the data shows, read from 100 sites on 9 November 2026.
Blocking is rare
Only 6 of 94 sites block one or more AI crawlers from their home page. The rest leave every AI crawler we checked free to read it.
Most say nothing about AI
21% of the readable files name an AI crawler at all. The others have no AI-specific rules, so AI crawlers follow the general rules for every bot.
Some refuse unknown bots instead
9 of the 100 sites refused our declared bot at the home page with a 403 or 429 reply. That kind of block happens in the firewall or CDN, not in robots.txt, so it does not show up in a robots.txt check.
What it means for you
If your own site is meant to be found in AI answers, check two places: robots.txt and your firewall or CDN bot settings. Our free AI crawler checker reads the first one in a few seconds.
The data
In charts.
- Meta-ExternalAgent33%
- CCBot33%
- Bytespider33%
- GPTBot11%
- ClaudeBot11%
- Google-Extended11%
- Applebot-Extended11%
- Amazonbot11%
- OAI-SearchBot00%
- ChatGPT-User00%
- Claude-SearchBot00%
- Claude-User00%
- PerplexityBot00%
- Perplexity-User00%
- Meta-ExternalFetcher00%
Method
How we checked.
- Sample: all 100 companies on the Forbes Cloud 100 (2025) list, with the website Forbes lists for each. Where Forbes left the website blank, we used the company's own domain.
- Checked on 9 November 2026. We fetched https://domain/robots.txt with a declared user agent: Mozilla/5.0 (compatible; KookiesResearch/1.0; +https://www.kookies.io/research).
- For each file we applied the robots.txt rules (longest match wins, Allow wins a tie) to the home page path / for 15 AI crawler and fetcher tokens: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, Amazonbot, CCBot and Bytespider.
- "Names an AI crawler" means the file has a User-agent line for one of those tokens.
Limits
What it does not show.
- We checked the home page path only. A site can still block AI crawlers from other folders.
- 6 sites did not return a readable robots.txt to our request. They are listed as not read.
- robots.txt is a request, not a lock. Some AI fetchers say they may not follow it, and firewalls can block bots that robots.txt allows.
- One check on one day. Files change.
- Rank
- Company
- Domain
- robots.txt
- AI crawlers blocked from the home page
- Names an AI crawler
Kookies, "Do top cloud companies block AI crawlers?", checked 9 November 2026. https://www.kookies.io/research/ai-crawlers-cloud-100| # | Company | robots.txt | AI crawlers blocked from the home page | Names an AI crawler |
|---|---|---|---|---|
| 1 | OpenAIopenai.com | Read | None | No |
| 2 | Anthropicanthropic.com | Read | None | No |
| 3 | Stripestripe.com | Read | None | No |
| 4 | Databricksdatabricks.com | Read | None | No |
| 5 | Canvacanva.com | Read | GPTBot, ClaudeBot, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider | Yes |
| 6 | Rampramp.com | Read | None | No |
| 7 | Navannavan.com | Read | None | Yes |
| 8 | Anyspherecursor.com | Read | None | No |
| 9 | Deeldeel.com | Read | None | Yes |
| 10 | Notionnotion.com | Read | Meta-ExternalAgent, Amazonbot | Yes |
| 11 | Grammarlygrammarly.com | Read | None | No |
| 12 | Celoniscelonis.com | Read | None | No |
| 13 | Grafana Labsgrafana.com | Read | None | No |
| 14 | Netskopenetskope.com | Read | None | No |
| 15 | Gustogusto.com | Could not read | ||
| 16 | Ripplingrippling.com | Read | None | No |
| 17 | Taniumtanium.com | Read | None | Yes |
| 18 | Miromiro.com | Read | None | No |
| 19 | Abnormal AIabnormal.ai | Read | None | No |
| 20 | Attentiveattentive.com | Read | None | No |
| 21 | Cohesitycohesity.com | Could not read | ||
| 22 | Arctic Wolfarcticwolf.com | Read | None | No |
| 23 | Flock Safetyflocksafety.com | Read | None | No |
| 24 | VAST Datavastdata.com | Read | None | No |
| 25 | AlphaSensealpha-sense.com | Read | None | No |
| 26 | Perplexity AIperplexity.ai | Read | None | No |
| 27 | Checkrcheckr.com | Read | None | No |
| 28 | OneTrustonetrust.com | Read | None | No |
| 29 | Plaidplaid.com | Read | None | No |
| 30 | Talkdesktalkdesk.com | Read | None | No |
| 31 | Cartacarta.com | Read | None | No |
| 32 | Postmanpostman.com | Read | Bytespider | Yes |
| 33 | Zohozoho.com | Read | None | No |
| 34 | Airtableairtable.com | Read | None | No |
| 35 | Dataikudataiku.com | Read | None | No |
| 36 | Fivetranfivetran.com | Read | None | No |
| 37 | Criblcribl.io | Read | None | No |
| 38 | Dialpaddialpad.com | Read | None | No |
| 39 | 1Password1password.com | Read | None | No |
| 40 | Checkout.comcheckout.com | Read | None | No |
| 41 | Automation Anywhereautomationanywhere.com | Read | None | No |
| 42 | Motivegomotive.com | Read | None | No |
| 43 | Vercelvercel.com | Read | None | No |
| 44 | Gleanglean.com | Read | None | Yes |
| 45 | Airwallexairwallex.com | Read | None | No |
| 46 | Clioclio.com | Read | None | No |
| 47 | NinjaOneninjaone.com | Read | None | Yes |
| 48 | Zapierzapier.com | Read | None | Yes |
| 49 | Lambdalambda.ai | Read | None | No |
| 50 | Brexbrex.com | Read | None | No |
| 51 | Snyksnyk.io | Read | None | No |
| 52 | Gonggong.io | Read | None | No |
| 53 | Intercomintercom.com | Read | None | No |
| 54 | Guildguild.com | Read | None | No |
| 55 | HiBobhibob.com | Read | None | Yes |
| 56 | Collibracollibra.com | Read | None | No |
| 57 | Abridgeabridge.com | Read | None | Yes |
| 58 | AppsFlyerappsflyer.com | Read | None | No |
| 59 | Calendlycalendly.com | Read | CCBot | Yes |
| 60 | Komodo Healthkomodohealth.com | Read | None | No |
| 61 | Workatoworkato.com | Read | None | No |
| 62 | Pendopendo.io | Read | None | No |
| 63 | Vantavanta.com | Read | None | No |
| 64 | Webflowwebflow.com | Could not read | ||
| 65 | Forterforter.com | Read | Meta-ExternalAgent, CCBot, Bytespider | Yes |
| 66 | Benchlingbenchling.com | Read | None | No |
| 67 | BrowserStackbrowserstack.com | Read | None | No |
| 68 | Midjourneymidjourney.com | Could not read | ||
| 69 | Personiopersonio.com | Could not read | ||
| 70 | ClickUpclickup.com | Read | None | No |
| 71 | Automatticautomattic.com | Read | None | No |
| 72 | ClickHouseclickhouse.com | Read | None | No |
| 73 | Axoniusaxonius.com | Could not read | ||
| 74 | Algoliaalgolia.com | Read | None | No |
| 75 | DeepLdeepl.com | Read | None | No |
| 76 | Hugging Facehuggingface.co | Read | None | No |
| 77 | Harnessharness.io | Read | None | No |
| 78 | Islandisland.io | Read | None | No |
| 79 | MaintainXgetmaintainx.com | Read | None | No |
| 80 | ElevenLabselevenlabs.io | Read | None | No |
| 81 | Cato Networkscatonetworks.com | Read | None | Yes |
| 82 | Cyeracyera.com | Read | None | Yes |
| 83 | Runwayrunwayml.com | Read | None | No |
| 84 | Fireworks AIfireworks.ai | Read | None | No |
| 85 | Odooodoo.com | Read | None | No |
| 86 | Together AItogether.ai | Read | Google-Extended | Yes |
| 87 | Dbt Labsgetdbt.com | Read | None | No |
| 88 | Fal AIfal.ai | Read | None | No |
| 89 | Mercormercor.com | Read | None | Yes |
| 90 | LaunchDarklylaunchdarkly.com | Read | None | No |
| 91 | Harveyharvey.ai | Read | None | No |
| 92 | Sierrasierra.ai | Read | None | No |
| 93 | Papaya Globalpapayaglobal.com | Read | None | Yes |
| 94 | Brightwheelmybrightwheel.com | Read | None | No |
| 95 | GlossGeniusglossgenius.com | Read | None | Yes |
| 96 | Clarotyclaroty.com | Read | None | No |
| 97 | Chainguardchainguard.dev | Read | None | Yes |
| 98 | Synthesiasynthesia.io | Read | None | No |
| 99 | Clayclay.com | Read | None | No |
| 100 | EvenUpevenuplaw.com | Read | None | No |
Next steps
Check your own site.
How many top cloud companies have an llms.txt?
How ready are top cloud companies for AI search?
Do top cloud companies show their prices?
How fast are top cloud company home pages?
Do pricing pages and articles score like the home page?
How long are the demo forms of top cloud companies?
Do top cloud companies send security headers?
How do top cloud companies look when shared?
What do top cloud sites load before you click?
How accessible are the home pages of top cloud companies?
FAQs
Questions about
this study.

The 100 companies on the Forbes Cloud 100 (2025) list, a yearly ranking of private cloud companies by Forbes, Bessemer Venture Partners and Salesforce Ventures. The data file lists every site.
We applied each site's robots.txt rules to the home page path for each crawler token, using the standard rules: the longest matching rule wins, and Allow wins a tie.
No. Allowing a crawler only means the site does not ask it to stay away. Whether an AI tool shows or cites a page depends on many other things.
Some sites refused our declared bot or did not answer in time. We list them as not read rather than guess.
Yes. Download the CSV and credit Kookies with a link to this page.
Next studies
Get the next
study first.
We publish a new study when the data is in. Leave your email and we send each one when it is out. Nothing else.
We send the next study the day it is out.


