FREEFirst project with us? Your landing page design is on the house.See the offer

Research · AI crawlers

Do top cloud companies block AI crawlers?

We read the robots.txt file of every company on the Forbes Cloud 100 (2025) list and checked the rules for 15 AI crawlers and fetchers.

Sample: Forbes Cloud 100 (2025)Checked 9 November 2026100 sites

Almost none of them block AI crawlers. Of the 94 companies whose robots.txt we could read, 94% let every AI crawler we checked read their home page. Only 1 blocks OpenAI's GPTBot and 1 blocks Anthropic's ClaudeBot. Most never mention AI crawlers at all: 21% name one in robots.txt.

94%let every AI crawler we checked read the home page
6of 94 block at least one AI crawler
1site blocks GPTBot (OpenAI training)
21%name an AI crawler in robots.txt at all

Findings

What we found.

4 things the data shows, read from 100 sites on 9 November 2026.

01

Blocking is rare

Only 6 of 94 sites block one or more AI crawlers from their home page. The rest leave every AI crawler we checked free to read it.

02

Most say nothing about AI

21% of the readable files name an AI crawler at all. The others have no AI-specific rules, so AI crawlers follow the general rules for every bot.

03

Some refuse unknown bots instead

9 of the 100 sites refused our declared bot at the home page with a 403 or 429 reply. That kind of block happens in the firewall or CDN, not in robots.txt, so it does not show up in a robots.txt check.

04

What it means for you

If your own site is meant to be found in AI answers, check two places: robots.txt and your firewall or CDN bot settings. Our free AI crawler checker reads the first one in a few seconds.

The data

In charts.

Sites that block each AI crawlerof 94 readable robots.txt files
  • Meta-ExternalAgent33%
  • CCBot33%
  • Bytespider33%
  • GPTBot11%
  • ClaudeBot11%
  • Google-Extended11%
  • Applebot-Extended11%
  • Amazonbot11%
  • OAI-SearchBot00%
  • ChatGPT-User00%
  • Claude-SearchBot00%
  • Claude-User00%
  • PerplexityBot00%
  • Perplexity-User00%
  • Meta-ExternalFetcher00%

Method

How we checked.

  1. Sample: all 100 companies on the Forbes Cloud 100 (2025) list, with the website Forbes lists for each. Where Forbes left the website blank, we used the company's own domain.
  2. Checked on 9 November 2026. We fetched https://domain/robots.txt with a declared user agent: Mozilla/5.0 (compatible; KookiesResearch/1.0; +https://www.kookies.io/research).
  3. For each file we applied the robots.txt rules (longest match wins, Allow wins a tie) to the home page path / for 15 AI crawler and fetcher tokens: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, Amazonbot, CCBot and Bytespider.
  4. "Names an AI crawler" means the file has a User-agent line for one of those tokens.

Limits

What it does not show.

  • We checked the home page path only. A site can still block AI crawlers from other folders.
  • 6 sites did not return a readable robots.txt to our request. They are listed as not read.
  • robots.txt is a request, not a lock. Some AI fetchers say they may not follow it, and firewalls can block bots that robots.txt allows.
  • One check on one day. Files change.

Data

Every site,
one row each.

Free to use under CC BY 4.0. Credit Kookies with a link to this page.

ai-crawlers-cloud-100.csv100 rows · 6 columns · CC BY 4.0
  • Rank
  • Company
  • Domain
  • robots.txt
  • AI crawlers blocked from the home page
  • Names an AI crawler
Download the CSV100 rows
Cite this studyKookies, "Do top cloud companies block AI crawlers?", checked 9 November 2026. https://www.kookies.io/research/ai-crawlers-cloud-100
#Companyrobots.txtAI crawlers blocked from the home pageNames an AI crawler
1OpenAIopenai.comReadNoneNo
2Anthropicanthropic.comReadNoneNo
3Stripestripe.comReadNoneNo
4Databricksdatabricks.comReadNoneNo
5Canvacanva.comReadGPTBot, ClaudeBot, Applebot-Extended, Meta-ExternalAgent, CCBot, BytespiderYes
6Rampramp.comReadNoneNo
7Navannavan.comReadNoneYes
8Anyspherecursor.comReadNoneNo
9Deeldeel.comReadNoneYes
10Notionnotion.comReadMeta-ExternalAgent, AmazonbotYes
11Grammarlygrammarly.comReadNoneNo
12Celoniscelonis.comReadNoneNo
13Grafana Labsgrafana.comReadNoneNo
14Netskopenetskope.comReadNoneNo
15Gustogusto.comCould not read
16Ripplingrippling.comReadNoneNo
17Taniumtanium.comReadNoneYes
18Miromiro.comReadNoneNo
19Abnormal AIabnormal.aiReadNoneNo
20Attentiveattentive.comReadNoneNo
21Cohesitycohesity.comCould not read
22Arctic Wolfarcticwolf.comReadNoneNo
23Flock Safetyflocksafety.comReadNoneNo
24VAST Datavastdata.comReadNoneNo
25AlphaSensealpha-sense.comReadNoneNo
26Perplexity AIperplexity.aiReadNoneNo
27Checkrcheckr.comReadNoneNo
28OneTrustonetrust.comReadNoneNo
29Plaidplaid.comReadNoneNo
30Talkdesktalkdesk.comReadNoneNo
31Cartacarta.comReadNoneNo
32Postmanpostman.comReadBytespiderYes
33Zohozoho.comReadNoneNo
34Airtableairtable.comReadNoneNo
35Dataikudataiku.comReadNoneNo
36Fivetranfivetran.comReadNoneNo
37Criblcribl.ioReadNoneNo
38Dialpaddialpad.comReadNoneNo
391Password1password.comReadNoneNo
40Checkout.comcheckout.comReadNoneNo
41Automation Anywhereautomationanywhere.comReadNoneNo
42Motivegomotive.comReadNoneNo
43Vercelvercel.comReadNoneNo
44Gleanglean.comReadNoneYes
45Airwallexairwallex.comReadNoneNo
46Clioclio.comReadNoneNo
47NinjaOneninjaone.comReadNoneYes
48Zapierzapier.comReadNoneYes
49Lambdalambda.aiReadNoneNo
50Brexbrex.comReadNoneNo
51Snyksnyk.ioReadNoneNo
52Gonggong.ioReadNoneNo
53Intercomintercom.comReadNoneNo
54Guildguild.comReadNoneNo
55HiBobhibob.comReadNoneYes
56Collibracollibra.comReadNoneNo
57Abridgeabridge.comReadNoneYes
58AppsFlyerappsflyer.comReadNoneNo
59Calendlycalendly.comReadCCBotYes
60Komodo Healthkomodohealth.comReadNoneNo
61Workatoworkato.comReadNoneNo
62Pendopendo.ioReadNoneNo
63Vantavanta.comReadNoneNo
64Webflowwebflow.comCould not read
65Forterforter.comReadMeta-ExternalAgent, CCBot, BytespiderYes
66Benchlingbenchling.comReadNoneNo
67BrowserStackbrowserstack.comReadNoneNo
68Midjourneymidjourney.comCould not read
69Personiopersonio.comCould not read
70ClickUpclickup.comReadNoneNo
71Automatticautomattic.comReadNoneNo
72ClickHouseclickhouse.comReadNoneNo
73Axoniusaxonius.comCould not read
74Algoliaalgolia.comReadNoneNo
75DeepLdeepl.comReadNoneNo
76Hugging Facehuggingface.coReadNoneNo
77Harnessharness.ioReadNoneNo
78Islandisland.ioReadNoneNo
79MaintainXgetmaintainx.comReadNoneNo
80ElevenLabselevenlabs.ioReadNoneNo
81Cato Networkscatonetworks.comReadNoneYes
82Cyeracyera.comReadNoneYes
83Runwayrunwayml.comReadNoneNo
84Fireworks AIfireworks.aiReadNoneNo
85Odooodoo.comReadNoneNo
86Together AItogether.aiReadGoogle-ExtendedYes
87Dbt Labsgetdbt.comReadNoneNo
88Fal AIfal.aiReadNoneNo
89Mercormercor.comReadNoneYes
90LaunchDarklylaunchdarkly.comReadNoneNo
91Harveyharvey.aiReadNoneNo
92Sierrasierra.aiReadNoneNo
93Papaya Globalpapayaglobal.comReadNoneYes
94Brightwheelmybrightwheel.comReadNoneNo
95GlossGeniusglossgenius.comReadNoneYes
96Clarotyclaroty.comReadNoneNo
97Chainguardchainguard.devReadNoneYes
98Synthesiasynthesia.ioReadNoneNo
99Clayclay.comReadNoneNo
100EvenUpevenuplaw.comReadNoneNo

FAQs

Questions about
this study.

Abdul Quadir
Still have a question?Ask us directly. We reply within a day.
Book a callPick a timehello@kookies.io

The 100 companies on the Forbes Cloud 100 (2025) list, a yearly ranking of private cloud companies by Forbes, Bessemer Venture Partners and Salesforce Ventures. The data file lists every site.

We applied each site's robots.txt rules to the home page path for each crawler token, using the standard rules: the longest matching rule wins, and Allow wins a tie.

No. Allowing a crawler only means the site does not ask it to stay away. Whether an AI tool shows or cites a page depends on many other things.

Some sites refused our declared bot or did not answer in time. We list them as not read rather than guess.

Yes. Download the CSV and credit Kookies with a link to this page.

Next studies

Get the next
study first.

We publish a new study when the data is in. Leave your email and we send each one when it is out. Nothing else.

One email per study. We do not share your email. Privacy policy

Your turn

Let's kook
what you krave.

Hi Kookies, I need a built on . I'd love to launch . Reach me at
or book a callSent. We reply within one working day.