Most data on websites blocking AI crawlers comes from robots.txt files, or from the biggest sites on the web. Neither shows what a local business's server actually sends back to a bot that says it is ChatGPT, Claude or Perplexity.
That matters because people now ask AI assistants for "a good chiropractor near me", and an assistant can only recommend a business whose site it can read, which is why AI crawler access is the first check in theStacc's AI Search Visibility module. So we asked the site directly. We requested 140,000 US local business homepages as nine bots and as a normal browser, from the same connection, then rechecked every refusal at least a day later. Here is what came back.
Sample: 140,000 US local business websites from Overture Maps Places, across 30 industries and all 50 states plus DC. After removing parked, for-sale and placeholder domains, 88,640 sites loaded normally and formed the live test. Overture's categories include a small number of non-local organisations.
Method: each homepage was requested twice as a desktop browser, once as Googlebot and once as each of eight AI bots from OpenAI, Anthropic and Perplexity. Every refusal was rechecked at least 24 hours later.
Dates: October 31 to November 2, 2026. Full method and limitations are below.
The 12 findings at a glance
- 1 in 33 US local business websites (3.0% of 83,504 eligible sites) refused requests identifying as an AI search bot that its own robots.txt allows, while serving Googlebot normally, according to a November 2026 theStacc test.
- 83.2% of those sites returned a block page identical to other sites' block pages, consistent with shared platform or hosting rules rather than owner choices.
- On one health-practice website platform, 98.1% of sites (949 of 967) refused requests identifying as GPTBot and all three Claude agents, although their robots.txt allows GPTBot and ClaudeBot by name.
- In robots.txt, owners block AI training far more than AI search: 3.2% block training bots only, and just 13 sites block search bots without also blocking training bots.
- 57% of small businesses say they restrict AI crawlers (Clutch), but only 7.1% of local business websites block any AI bot in robots.txt.
- 79.0% of Wix sites challenged requests identifying as an AI search bot, and most also refused requests identifying as Googlebot.
- Counting automated bot challenges, up to 13.6% of sites turned away requests identifying as an AI search bot their robots.txt allows.
- 78.8% of sites served all three AI search bots normally.
- 9.6% of homepages have fewer than 100 words of text before JavaScript runs.
- 24.3% of sites have an llms.txt file, and 65.6% of those look auto-generated.
- 1.2% of sites return a server error for robots.txt, which the robots.txt standard treats as "complete disallow".
- 98.5% of Square Online and 94.6% of Weebly local business sites refused requests identifying as AI training crawlers but served every AI search bot, according to a November 2026 theStacc test of 88,640 US websites.
AI search bot: a crawler that finds pages for an AI assistant's search answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot).
AI training bot: a crawler that collects pages for model training (GPTBot, ClaudeBot).
User agent: a fetcher that loads a page when a person asks an assistant a question (ChatGPT-User, Claude-User, Perplexity-User).
Refused while serving Googlebot: the AI bot got a block (most often HTTP 403) on every check, while Googlebot and a browser loaded the page. Our strictest measure.
Turned away: any refusal, including automated "checking your request" challenges (HTTP 429). Our broadest measure.
theStacc is an AI marketing platform, so we have a commercial interest in how businesses think about AI search. Every figure on this page comes from our own crawl data and analysis scripts, the definitions are published, and the limitations section lists what this data cannot tell you. No individual business is named on this page or in the downloadable data.
Quick answers
How many local business websites block AI crawlers?
7.1% of US local business websites (6,965 of 98,195) block at least one AI crawler in robots.txt, according to a November 2026 theStacc analysis. Separately, 3.0% of sites whose robots.txt allows Googlebot and an AI search bot (2,523 of 83,504) refused requests identifying as that AI search bot while serving Googlebot. See finding 5 and finding 1.
How many websites block GPTBot?
5.6% of US local business websites (5,473 of 98,195) block GPTBot in robots.txt, according to a November 2026 theStacc analysis. Bytespider is blocked more often, at 6.5%. See finding 5.
Does Wix block AI crawlers?
79.0% of Wix local business sites (7,136 of 9,035) challenged requests identifying as an AI search bot in a November 2026 theStacc test, and 68.5% of those also refused requests identifying as Googlebot. See finding 6.
What share of small business websites have an llms.txt file?
24.3% of reachable US local business websites (21,898 of 90,103) serve an llms.txt file, and 65.6% of those look auto-generated, according to a November 2026 theStacc test. See finding 10.
Do small businesses block ChatGPT search on purpose?
Rarely. Of 98,195 US local business websites, 2.4% block an AI search bot in robots.txt and only 13 block AI search bots without also blocking AI training bots, a November 2026 theStacc analysis found. See finding 4.
Nine bots, three jobs: why the difference matters
AI companies run separate bots for separate jobs, and a site can treat each one differently. OpenAI says GPTBot is used to make its "generative AI foundation models more useful and safe", OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features", and ChatGPT-User handles actions a user asks for. It adds that "each setting is independent of the others" (OpenAI). Anthropic draws the same lines for ClaudeBot, Claude-SearchBot and Claude-User (Anthropic).
The practical upshot: a business can keep its content out of AI training and still appear in AI search answers. Refusing a search bot or a user agent is what keeps a site out of those answers. That is why this study centres on the three search bots, with the training bots, user agents and Googlebot tested alongside them.
1. 1 in 33 sites refuses requests identifying as AI search bots it allows
Of 83,504 US local business websites whose robots.txt allows Googlebot and at least one AI search bot, 2,523 (3.0%) refused requests identifying as that AI search bot while serving Googlebot and a desktop browser normally.
robots.txt tells crawlers what they may visit, but a firewall or server rule can still turn one away. Preferly's October 2026 study of 3,693 local business sites found 86% fully open to AI crawlers and was candid about the gap: "We couldn't measure that without pretending to be that crawler." We did exactly that, for every bot, on every site.
Every one of these refusals was a non-429 response (most often HTTP 403) that held on every check, including the recheck. By bot, 1.8% refused Claude-SearchBot, 1.4% refused OAI-SearchBot and 1.1% refused PerplexityBot. Excluding the single platform in finding 3, the Claude-SearchBot rate falls to 0.7%, the lowest of the three.
These sites served our Googlebot request, so they were not checking where bot traffic comes from. They refused the AI bot by name. That makes 3.0% the most conservative figure in this study, and the one least likely to be an artefact of the test.
Your robots.txt can say "come in" while your server says no. Only a live test shows which one an AI search bot hears.
2. Most lockouts look like shared set-ups, not owner choices
Of the 2,523 sites that refused an allowed AI search bot while serving Googlebot, 2,100 (83.2%) returned a block response identical to at least one other site's. 1,748 (69.3%) share it with 10 or more sites, and the 10 largest groups account for 1,479 sites (58.6%).
If owners were refusing AI bots one by one, the block pages would differ from site to site. They mostly don't. We grouped sites by the status code each AI bot received, the server type, and the size and title of the block page.
| Group size | What the bots received | Mostly |
|---|---|---|
| 949 sites | 403 to GPTBot, ClaudeBot, Claude-SearchBot and Claude-User | Chiropractors, dentists, vets, optometrists |
| 113 sites | Identical 403 page to GPTBot, OAI-SearchBot, ClaudeBot and Claude-SearchBot | Physical therapy |
| 114 sites (3 response groups) | "405 Not allowed." to most bots | Nail salons |
| 46 sites | Block page to PerplexityBot only | Hotels |
| 46 sites | 403 to GPTBot, OAI-SearchBot and Claude-SearchBot | Car dealers |
| 42 sites | 403 to GPTBot, ClaudeBot and PerplexityBot | Optometrists |
Identical responses across dozens or hundreds of unrelated businesses in the same industry are consistent with a platform, host or agency applying one rule to every site it runs. A generic server error page can occasionally match by coincidence, so we call these shared set-ups, not proven vendor decisions.
The AI setting that matters may sit with whoever built or hosts your site. Ask them which AI bots they allow.
3. On one health-practice platform, 98% of sites refuse requests identifying as GPTBot and Claude
Of 967 US local business websites sharing one robots.txt template, 949 (98.1%) refused requests identifying as GPTBot, ClaudeBot, Claude-SearchBot and Claude-User, while requests identifying as OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User and Googlebot were served on all 967.
The template lists GPTBot, ClaudeBot and other crawlers by name with "Allow: /", and lets the remaining bots through its general rule. These 949 sites make up 37.6% of the lockouts in finding 1: chiropractors (377), dentists (193), veterinarians (183), optometrists (123) and 73 other sites, mostly other health practices.
The refusals came from the content delivery network in front of the sites. We have not established who configured the rule, so we do not name the platform. This group also explains why chiropractors top the "refused while serving Googlebot" column of the industry table. Without it, nail salons lead that column at 6.9%.
If your practice website runs on a shared platform, check what Claude and GPTBot receive. The robots.txt file alone will not tell you.
4. Owners block AI training, not AI search
Of 98,195 US local business websites, 3.2% (3,150) block AI training bots only in robots.txt, 2.4% (2,370) block training and search bots, and just 13 block AI search bots without also blocking training bots. 94.4% block neither.
When a robots.txt file restricts AI search bots, it nearly always restricts training bots too. Almost nobody blocks AI search on its own. The data suggests local businesses rarely set out to disappear from AI answers.
5. Most owners say they block AI. Their robots.txt says otherwise
Only 7.1% of US local business websites (6,965 of 98,195) block any AI bot in robots.txt. The most-blocked are Bytespider (ByteDance) at 6.5%, meta-externalagent (Meta) at 5.6% and GPTBot at 5.6%.
Compare that with what owners report. In a Clutch survey of small businesses published in September 2025, 57% said they "choose to restrict access" for AI crawlers.
The two numbers measure different things (a survey answer and a file on the website), and the samples differ, so the gap is indicative, not exact. Still, it suggests many owners who believe they restrict AI crawlers have not done it in robots.txt, or rely on their platform or host to do it for them.
6. Wix sites challenge requests identifying as AI bots, and most refuse Googlebot requests too
79.0% of Wix sites (7,136 of 9,035) answered a request identifying as an AI search bot with an automated challenge page ("Checking Your Request", HTTP 429). On 68.5% of those sites (4,891), the request identifying as Googlebot was refused as well. A desktop browser on the same connection was served normally.
Because most Googlebot requests were refused too, this looks like general protection against requests that claim to be a bot, not a rule aimed at AI. Real crawlers that the network can verify may be treated differently from ours. We report it because Wix sites make up a large share of the wider figure in finding 7.
7. Counting bot challenges, up to 13.6% turn away requests identifying as AI search bots
12,058 of 88,640 US local business websites (13.6%) turned away a request identifying as an AI search bot that their robots.txt allows, on both passes. 61.3% of those sent only HTTP 429 challenges, and 7,136 were Wix sites. Outside Wix, the rate is 6.2% (4,922 of 79,605).
We treat 13.6% as an upper bound. Many of these challenges would likely not apply to the real crawlers, which some networks recognise and let through. The 3.0% in finding 1 is the figure we stand behind as a refusal by bot name.
8. Nearly 4 in 5 local sites are fully open to AI search
69,821 of 88,640 US local business websites (78.8%) served OAI-SearchBot, Claude-SearchBot and PerplexityBot normally. For comparison, Preferly's October 2026 study, which checked 3,693 local homepages once with its own crawler, found 86% fully open to AI crawlers.
For AI search to recommend a business, its bots first have to load the site, and most can. For the majority of local businesses, the open question is less about access and more about whether the site gives an AI model enough to recommend them.
9. 1 in 10 homepages shows almost no text before JavaScript runs
9.6% of US local business homepages (8,547 of 88,640) contain fewer than 100 words of visible text in the HTML a bot receives, before any JavaScript runs.
A crawler that does not run JavaScript sees only the text in a page's HTML. In tests published by Vercel in December 2024, "none of the major AI crawlers currently render JavaScript."
A homepage that builds its text in the browser can look complete to a visitor and nearly empty to a crawler that does not run JavaScript. Our AI visibility checker shows what a crawler sees on a given page.
10. A quarter have llms.txt, mostly made by their website builder
21,898 of 90,103 reachable US local business websites (24.3%) serve an llms.txt file. Of those, 65.6% look auto-generated, and 56.9% are on website builders that create the file automatically.
llms.txt is a proposed file that summarises a site for AI models. Among the largest websites it is still rare: Rankability's September 2026 scan found a confirmed file on 9.3% of the top 1,000 domains.
Among local businesses, the file is spreading through their platforms rather than through owners writing one. If you want yours to say something specific, our llms.txt generator is a starting point.
11. 1.2% return a robots.txt server error, which the standard treats as "stay out"
1,207 of 99,402 US local business websites (1.2%) returned a server error (HTTP 5xx) for robots.txt.
Under the robots.txt standard, RFC 9309, if robots.txt is unreachable because of a server error, "the crawler MUST assume complete disallow." If the error lasts a long time (the standard gives 30 days as an example), a crawler may instead treat the file as unavailable or use a cached copy.
Our result is a single snapshot, so some of these errors may be brief. While one lasts, a crawler that follows the standard should treat the whole site as off limits, for AI bots and search engines alike. Our robots.txt checker shows what a site returns.
12. Square's builders refuse requests identifying as AI training bots but serve AI search bots
852 of 865 Square Online sites (98.5%) and 2,220 of 2,346 Weebly sites (94.6%) refused requests identifying as GPTBot or ClaudeBot while serving every AI search bot normally. Weebly is part of Square, and most Square Online sites also carry Weebly markers, so the two groups overlap. On Wix, the same pattern appears on 4.6% of sites.
This is the split OpenAI and Anthropic describe: keep content out of model training while staying visible in AI search. On these two builders, the platform appears to apply it to nearly every site.
Which industries are most locked out of AI search?
The first rate counts any refusal of an allowed AI search bot (finding 7). The second counts only refusals while Googlebot was served (finding 1). Chiropractors, dentists, veterinarians and optometrists are raised by the single platform in finding 3.
| Industry | Turned away an AI search bot | Of sites | Refused while serving Googlebot | Of sites |
|---|---|---|---|---|
| Chiropractors | 23.9% (763) | 3,187 | 14.4% (430) | 2,995 |
| Physical therapy | 21.4% (699) | 3,261 | 5.9% (182) | 3,079 |
| Hair salons and barbers | 21.0% (485) | 2,311 | 1.0% (22) | 2,197 |
| Gyms and fitness | 20.1% (606) | 3,016 | 1.3% (36) | 2,866 |
| Child care | 18.5% (563) | 3,040 | 1.5% (42) | 2,885 |
| Med spas and day spas | 17.6% (490) | 2,789 | 0.9% (24) | 2,642 |
| Bakeries | 17.4% (481) | 2,763 | 1.0% (25) | 2,611 |
| Auto repair | 14.7% (377) | 2,560 | 4.5% (107) | 2,401 |
| Bars | 14.6% (393) | 2,683 | 1.2% (30) | 2,535 |
| Medical clinics | 14.5% (457) | 3,151 | 2.8% (82) | 2,974 |
| Real estate | 14.2% (404) | 2,840 | 4.1% (112) | 2,717 |
| Cafes and coffee shops | 13.8% (379) | 2,742 | 1.0% (26) | 2,610 |
| Restaurants | 13.4% (337) | 2,520 | 1.1% (27) | 2,355 |
| Nail salons | 13.3% (332) | 2,491 | 6.9% (160) | 2,320 |
| Optometrists | 12.6% (428) | 3,397 | 5.4% (178) | 3,274 |
| General contractors | 12.6% (394) | 3,129 | 1.2% (34) | 2,902 |
| Dentists | 12.5% (447) | 3,563 | 7.1% (237) | 3,350 |
| Veterinarians | 12.5% (476) | 3,802 | 5.6% (205) | 3,674 |
| Hotels and motels | 12.5% (330) | 2,648 | 3.5% (87) | 2,469 |
| Landscaping | 12.3% (367) | 2,977 | 1.5% (43) | 2,786 |
| Cleaning services | 11.4% (336) | 2,935 | 1.6% (43) | 2,721 |
| Insurance agencies | 11.2% (297) | 2,652 | 1.7% (43) | 2,515 |
| Electricians | 11.2% (334) | 2,995 | 1.5% (41) | 2,787 |
| Law firms | 10.9% (335) | 3,079 | 2.3% (65) | 2,860 |
| Pest control | 9.6% (306) | 3,183 | 1.4% (42) | 2,967 |
| Plumbers | 9.0% (285) | 3,151 | 1.2% (35) | 2,938 |
| Roofers | 8.7% (287) | 3,282 | 1.3% (39) | 3,008 |
| HVAC | 8.5% (279) | 3,288 | 1.4% (43) | 3,097 |
| Accounting and tax | 8.3% (254) | 3,044 | 1.1% (31) | 2,879 |
| Car dealers | 6.3% (137) | 2,161 | 2.5% (52) | 2,090 |
Which states are most locked out of AI search?
The range is fairly narrow: from 11.3% of sites in Tennessee to 19.9% in Alaska turned away an allowed AI search bot (any refusal, including challenges). Every state has at least 565 sites.
| State | Turned away an AI search bot | Sites |
|---|---|---|
| Alaska | 19.9% (114) | 572 |
| Vermont | 17.2% (97) | 565 |
| Wyoming | 17.1% (98) | 573 |
| Maine | 16.9% (126) | 746 |
| South Carolina | 16.6% (245) | 1,476 |
| Montana | 16.4% (123) | 751 |
| Wisconsin | 15.8% (272) | 1,717 |
| Connecticut | 15.8% (205) | 1,298 |
| New Mexico | 15.3% (116) | 759 |
| Kansas | 15.2% (156) | 1,024 |
| North Carolina | 15.2% (379) | 2,492 |
| South Dakota | 15.1% (98) | 647 |
| North Dakota | 15.0% (91) | 606 |
| Ohio | 15.0% (363) | 2,421 |
| Indiana | 14.8% (236) | 1,600 |
| Iowa | 14.7% (158) | 1,074 |
| New Hampshire | 14.6% (121) | 829 |
| Rhode Island | 14.6% (104) | 714 |
| Missouri | 14.5% (232) | 1,602 |
| Oregon | 14.4% (196) | 1,359 |
| Minnesota | 14.4% (226) | 1,568 |
| Colorado | 13.9% (265) | 1,903 |
| Georgia | 13.9% (341) | 2,449 |
| Mississippi | 13.9% (103) | 743 |
| Oklahoma | 13.8% (148) | 1,074 |
| Michigan | 13.8% (327) | 2,375 |
| Virginia | 13.7% (296) | 2,154 |
| Idaho | 13.7% (121) | 881 |
| Illinois | 13.6% (399) | 2,932 |
| Hawaii | 13.5% (93) | 689 |
| Massachusetts | 13.5% (299) | 2,220 |
| Delaware | 13.4% (90) | 671 |
| Washington | 13.4% (279) | 2,089 |
| Arkansas | 13.3% (120) | 899 |
| Kentucky | 13.3% (145) | 1,091 |
| District of Columbia | 13.2% (78) | 592 |
| Texas | 13.2% (796) | 6,051 |
| Maryland | 13.1% (218) | 1,664 |
| New Jersey | 13.0% (330) | 2,538 |
| West Virginia | 12.9% (84) | 651 |
| Pennsylvania | 12.9% (384) | 2,983 |
| Alabama | 12.7% (146) | 1,146 |
| California | 12.6% (1,072) | 8,478 |
| New York | 12.4% (528) | 4,245 |
| Florida | 12.4% (736) | 5,946 |
| Louisiana | 12.2% (148) | 1,218 |
| Nebraska | 11.9% (103) | 866 |
| Nevada | 11.8% (122) | 1,035 |
| Utah | 11.4% (128) | 1,120 |
| Arizona | 11.4% (213) | 1,867 |
| Tennessee | 11.3% (190) | 1,677 |
How to check whether AI search can read your website
The findings point to five checks any local business can run. None needs a developer to start.
- 1Test what the bots receive. Run your homepage through the AI visibility checker. robots.txt alone does not show refusals from your server or firewall (findings 1 to 3).
- 2Read your robots.txt. Use the robots.txt checker to see which AI bots it allows and confirm it does not return a server error (finding 11).
- 3Separate training from search. If you want to stay out of AI training, block GPTBot and ClaudeBot, not OAI-SearchBot, Claude-SearchBot or PerplexityBot (findings 4 and 12).
- 4Ask your platform or host. Ask which AI bots they allow or challenge by default, and whether you can change it (findings 2, 3 and 6).
- 5Put your key text in the HTML. Make sure your services, location and contact details appear without JavaScript (finding 9).
Want AI search to have something worth reading on your site?
Once the bots can get in, content does the rest. theStacc publishes local blog content and Google Business Profile posts for your business. Compare plans first if you prefer.
What this data cannot tell you
Our requests named each bot but came from our servers. Some networks verify crawlers by IP address and may
treat the real GPTBot or Claude-SearchBot differently. We report "refused a request identifying as X", not "blocked OpenAI"
or "blocked Anthropic".
Challenges are not the same as refusals by name. 61.3% of the sites in finding 7 sent only automated HTTP 429
challenges, mostly on Wix, and many also refused requests identifying as Googlebot. That is why finding 1 uses a stricter
definition.
Not every refusal is a firewall product. Only 17.2% of the sites in finding 1 (433 of 2,523) returned a
recognised firewall or CDN block page; most refusals were plain 403 or 405 responses from the server, so we say "the server
refused" rather than "the firewall blocked".
Homepages only, one crawl window. We tested each homepage, robots.txt and llms.txt between October 31 and
November 2, 2026. Inner pages and later changes are not covered.
Shared set-ups are inferred. Identical block responses suggest a common platform or host, but we have not
confirmed who configured any block.
Sample. We drew 140,000 sites and targeted 100,000 usable results; 88,640 met the test conditions. Overture
Maps categories include a small number of non-local organisations. Industry and state rates are unweighted; national rates
weighted by state differ from the unweighted ones by less than one percentage point.
Method
- Sample. US places from Overture Maps Places (release 2026-10-21) with a website, a confidence score of 0.8 or higher and one of 30 local industries. We removed social, directory and booking domains and any domain listed at five or more US places (chains), kept one site per domain, and drew 140,000 sites with at least 700 per state.
- Requests. For each site: robots.txt (as our research crawler, theStaccResearch), the homepage as Chrome, the homepage once as each bot that robots.txt allows (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User and Googlebot) in random order, the homepage as Chrome again, then llms.txt. Bots that robots.txt disallows were never requested. Requests were spaced 1.5 to 3 seconds apart per site.
- Valid sites. Of 93,253 sites where both browser requests loaded and the rate-limit check passed, we removed 4,613 parked, for-sale, coming-soon and suspended domains (4,948 across the full sample) using rules applied to the stored responses, leaving 88,640 valid sites.
- Refused. A bot was refused if both browser requests loaded but the bot received a 401, 403, 406, 429, 451 or 503 status, a challenge page the browser did not get, or a different error than the browser. Every refusal was rechecked at least 24 hours later with a fresh bot order; a refusal counts only if the recheck confirmed it.
- Refused while serving Googlebot (finding 1). Among sites whose robots.txt allows Googlebot and an AI search bot: the AI search bot got a non-429 refusal in every valid observation while Googlebot and both browser requests were served. A small number of sites rate-limited on the first pass have a single observation.
- Shared set-ups (finding 2). Sites from finding 1 grouped by identical block responses: the status code each AI bot received, the server type, and the size and title of the block page.
- Analysis and AI use. Counts and percentages from scripts run on the stored responses; no figure was estimated or modelled. Claude (Anthropic) helped write the crawler and analysis scripts and draft this page. An independent recount by a separate process matched every key figure, and a person at theStacc reviewed the page.
The bottom line
Most local business websites are open to AI search, and very few owners block AI search bots on purpose. The lockouts we found cluster in shared website platforms, hosts and builders, often on sites whose robots.txt invites the very bot being refused. If AI search matters to your business, test what your homepage returns to these bots, then ask whoever runs your site which AI settings they apply.
What surprised you most in this data? If your website builder or host made an AI setting for you, we would like to hear about it.
Cite this study
theStacc (2026). The AI Lockout Index: a test of 88,640 US local business websites. https://thestacc.com/research/ai-crawler-blocking-statistics/
Data, charts and embed codes
- Main results (CSV): refusal rates by bot, industry, state, website builder and CDN, plus robots.txt and llms.txt figures.
- Additional figures (CSV): the finding 1 rates (overall, by bot, builder, CDN and industry) and the context figures used in findings 6, 7, 10 and 12.
- Shared set-ups summary (CSV): the finding 2 figures.
- Media pack (ZIP): every chart in high resolution (PNG and SVG), the summary infographic and a README.
No site-level data is published. You're free to use our charts and data with a link to this page.
Embed the main chart:
<a href="https://thestacc.com/research/ai-crawler-blocking-statistics/"><img src="https://thestacc.com/research/ai-lockout-index/ai-lockout-index-01-1-in-33-refuse-allowed-ai-search-bots.webp" alt="3.0% of local sites refused requests identifying as allowed AI search bots; Claude-SearchBot 1.8%, OAI-SearchBot 1.4%" width="1200"></a>
<p>Source: <a href="https://thestacc.com/research/ai-crawler-blocking-statistics/">The AI Lockout Index by theStacc</a></p>
Data & sources
- [01]theStacc crawl of 140,000 US local business websites, October 31 to November 2, 2026, with a recheck pass on November 2, 2026. 88,640 valid sites after exclusions.
- [02]Sample frame: Overture Maps Places, release 2026-10-21.0.
- [03]OpenAI: Overview of OpenAI crawlers
- [04]Anthropic: crawlers and how site owners can control them
- [05]Preferly: AI crawler study, October 22, 2026
- [06]Clutch: press release on small businesses and AI crawlers, September 24, 2025
- [07]Vercel: The rise of the AI crawler, December 17, 2024
- [08]RFC 9309: Robots Exclusion Protocol, September 2022
- [09]Rankability: llms.txt adoption tracker, scan of September 18, 2026
theStacc is an AI marketing platform for small businesses, agencies and multi-location brands. It learns a business once, then plans, writes and publishes search-led blog content, Google Business Profile posts and review replies, and social media for Instagram, Facebook, LinkedIn and X from one calendar, with manual, review-first or autopilot control for each channel. It shows where a business stands on Google Maps and in AI answers from ChatGPT, Gemini, Google AI Mode and Perplexity, and drafts the fixes. It includes analytics across Search Console, GA4 and AI-assistant visits, a site auditor, an AI assistant and agent access for Claude and other MCP clients. Google and Meta Ads, an AI website builder and AI video are part of the platform. theStacc also offers managed marketing services. It is built by FSG Agentic Solutions Private Limited in Jaipur, India.
