If you sell a service and you want to be recommended, don't block the AI crawlers that answer questions. Block the ones that train models if you want — those are different bots with different names, and confusing the two is the expensive mistake. GPTBot trains. OAI-SearchBot decides whether ChatGPT can pull up your page when a prospect asks it to recommend someone like you. Blocking the first costs you nothing today. Blocking the second removes you from the answer.
That distinction is about to matter to a lot more sites, because Cloudflare is changing what happens by default.
What actually changes on September 15, 2026
Cloudflare is retiring its one-size-fits-all "Block AI Bots" switch and replacing it with three separate controls, split by what the bot is for:
- Search — indexing your content so it can be surfaced in an answer or a result
- Agent — real-time automation acting on behalf of a user
- Training — collecting data to build models
For new domains onboarded after September 15, pages that display ads default to blocking Training and Agent while leaving Search allowed. Cloudflare's reasoning, in their own announcement, is that an ad is a signal the owner meant a human to land there. It applies to new domains and new sites added by existing customers, and existing customers can opt out through Security settings before the date.
Here's the part worth sitting with. Cloudflare sits in front of a large share of global web traffic, and AI-related crawlers went from about 22% of crawler requests in spring 2025 to roughly 52% by June 2026. More than half of what's hitting your site now isn't a search engine in the old sense. The infrastructure layer is starting to make a policy decision on your behalf, and the default it picks is the one most sites will live with forever.
So the question stops being philosophical. It's a configuration you either chose or inherited.
The six bots you're actually deciding about
Most of the confusion comes from people talking about "AI crawlers" as one thing. They aren't. Both OpenAI and Anthropic deliberately split their crawlers so site owners can make granular calls, and they document it publicly.
From OpenAI's bot documentation:
- GPTBot — crawls content that may be used to train foundation models
- OAI-SearchBot — indexes sites to surface them in ChatGPT's search features
- ChatGPT-User — fetches a page live when a user's question sends ChatGPT to it
- OAI-AdsBot — validates the safety of pages submitted as ads, not used for training
Anthropic mirrors it: ClaudeBot for training, Claude-SearchBot for search indexing, Claude-User for user-directed fetches.
Read that list again with a service business in mind. Only one of those bots — GPTBot, plus ClaudeBot on the other side — is doing the thing publishers are angry about. The rest are the machinery that puts you in front of a buyer.
When someone types "block AI crawlers" into a plugin settings page, they are usually thinking about GPTBot and unknowingly turning off all six.
Does blocking actually cost you citations?
This is where I'd rather show you the disagreement than pretend it's settled, because both findings are real and they explain each other.
- One analysis of roughly 4 million citations across thousands of prompts found that publishers blocking AI crawlers via robots.txt still showed up in the large majority of AI-generated citations. Among sites blocking ChatGPT-User, about 70% still appeared. Among sites blocking Google-Extended, over 92% did.
- A separate analysis found the opposite-looking result: domains blocking GPTBot earned almost no ChatGPT citations relative to the Google rankings they held, compared to domains that allowed it.
- A working paper on traffic found sites that blocked LLM crawlers in robots.txt lost roughly 7% of weekly traffic within six weeks.
These aren't contradictory once you separate being mentioned from being the source. A model can talk about a well-known brand from training memory and from what other sites say about it. That's why the big publishers still appear. What blocking removes is your page as the retrievable, linkable source — so the citation goes to the directory, the review site, or the competitor's comparison post that wrote about you instead.
If you're The New York Times, being talked about is enough and licensing revenue is real. If you're a roofing company in Plano, being talked about doesn't exist yet. You need the retrieval, because nobody is writing about you at scale for the model to remember.
The blocking calculus is a publisher's calculus, and most businesses copying it aren't publishers.
The failure we find most often on client audits
It's almost never a deliberate policy. It's a toggle.
The pattern goes like this: a site has a clean robots.txt that allows everything, so the owner assumes access is fine. Meanwhile a security plugin, a managed WordPress host, or a Cloudflare setting is blocking bots at the edge — before the request ever gets far enough to care what robots.txt says. Nothing in the SEO tooling flags it. Rankings look normal, because Googlebot is on an allowlist. The site is simply absent from AI answers and nobody can say why.
The second failure is broad pattern matching. Someone writes a firewall rule against a user-agent substring to catch AI bots and it also catches a legitimate search crawler. That one shows up in your analytics eventually, but "eventually" is usually a few weeks of lost traffic.
Both are found in about twenty minutes. Read the public file, then read the edge config, then confirm they agree. If robots.txt and your CDN disagree, the CDN wins and you have a problem.
For the record, our own file at vixi.agency/robots.txt names every one of these crawlers explicitly with Allow: / rather than leaving them to a default — including Claude-SearchBot, which we added while writing this. Being unambiguous costs nothing and removes the "did someone turn this off" question permanently.
So what should you actually do?
Decide by bot category, not by company, and not by headline.
- Allow the search and answer bots. OAI-SearchBot, Claude-SearchBot, PerplexityBot, and the user-triggered fetchers. These are how you get recommended. There's no upside to closing them unless you have a licensing deal that requires it.
- Decide on training bots on the merits. GPTBot and ClaudeBot. If your content is a genuine asset — original research, a large archive, journalism, anything you'd license — blocking is a defensible business position, and Cloudflare's pay-per-crawl work is aimed squarely at making that position pay. If your content is marketing for a service you sell, blocking training buys you nothing and costs you presence in the models people will ask next year.
- Leave Google-Extended alone unless you've thought about it. It governs Gemini and AI grounding, not Google Search rankings. Blocking it does not hurt your organic positions, but it does take you out of a growing answer surface.
- Never write a broad user-agent block. Name the bots. Specificity is the entire safety mechanism.
- Re-check Cloudflare before September 15. Especially if you enabled the old blanket toggle at some point and forgot. Existing customers can opt out of the new defaults in Security settings, and new sites you add after that date inherit them.
Access is table stakes, though, not a strategy. Opening the crawlers gets you eligible to be cited — it doesn't get you cited. If you're already open and still invisible in AI answers, the problem is entity clarity and content structure, and that's a different fix. We wrote about the diagnostic for that in why your content gets crawled but never cited, and about the publishing cadence that earns citations in daily AI content that gets cited. If you want the broader technical pass, the 2026 technical SEO audit checklist covers the rest of the crawl layer, and local businesses should pair it with local SEO for Dallas in 2026.
The short version
Blocking AI crawlers made sense as a publisher's stand against uncompensated training. It got copied by thousands of businesses that have nothing to license and everything to gain from being retrievable, using a single switch that never distinguished between the two.
Now the switch is being split into three, the defaults are moving, and more than half of your crawler traffic is AI. Spend twenty minutes finding out what your site currently says — the public file and the edge config both — and then make the call on purpose instead of inheriting it.
If you'd rather have someone else run that pass across your site, your CDN, and your AI answer visibility at the same time, get a free automation audit. We'll tell you exactly which crawlers you're turning away and what it's costing you in the answer layer.