Polski
Technical SEO · News

Cloudflare blocks AI bots from 15 September: check you are not blocking Googlebot

From 15 September 2026 Cloudflare blocks AI crawlers by default on sites that show ads. The catch is that Googlebot is a multi-purpose bot and may fall under the same block. I explain what is changing, who is affected and what to check in the dashboard before the deadline.

Sebastian Baryś
Sebastian Baryś
31 July 2026 13 min read

Introduction

English version of an article first published in Polish on 31 July 2026. Read the Polish original.

On 15 September 2026 Cloudflare changes its default settings and starts blocking some AI crawlers on sites that show ads. It sounds like a topic for big publishers, and partly it is. Except that the new rules will also cover every account on the free plan, which means a large share of Polish business websites, often set up by someone else and left untouched for years.

The most interesting part, though, is not the blocking itself but the way Cloudflare classifies bots. Instead of a single "let AI in or not" switch, there are now three categories of use. And bots that do several things at once, like Googlebot, are judged on all of their behaviours, with the most restrictive rule winning. That is where the risk sits: someone who once clicked "block AI bots" in good faith may, after the deadline, cut off a search engine crawler.

I break it down piece by piece: what exactly is changing, who it covers, where to find it in the dashboard and how to make the decision yourself instead of leaving it to the defaults.

You will learn:

  • What exactly changes at Cloudflare on 15 September 2026 and since when the new rules have been available
  • How the three bot categories differ: Search, Agent and Training
  • Why the old "Block AI bots" switch may block Googlebot after the deadline
  • Who the new defaults will cover and who keeps their own settings
  • What Pay Per Use is and whether you can realistically earn anything from it
  • A concrete checklist to work through in the dashboard before 15 September

What exactly Cloudflare is changing

On 1 July 2026 Cloudflare gave all customers a new way of managing crawlers: not by the bot's name, but by what the bot uses the fetched content for. At the same time it announced the date from which the default settings will change: 15 September 2026.

From that day, for new domains joining the Cloudflare network, the Training and Agent categories will be blocked by default on pages that show ads, while the Search category will remain allowed. Multi-purpose bots, which combine search with training, will from the same date be judged on all of their behaviours.

The "on pages with ads" condition matters here and is easy to miss. Cloudflare's logic is simple: if you monetise content with ads, every fetch by a model that does not send a user back to you is a pure loss. If you have no ads, the default blocking does not apply to you. That does not mean the topic passes you by, because the interpretation of settings you already have switched on is changing too.

Timeline of the changes

  1. 1 July 2026

    New categories for everyone

    Search, Agent and Training as separate switches in the dashboard

  2. until 14 September 2026

    Time to make your own decision

    settings can be changed in the dashboard at any time

  3. 15 September 2026

    New default settings

    Agent and Training blocked on pages with ads

There are two and a half months between the announcement and the change taking effect. That is all the time we have been given to decide.

Three bot categories instead of one switch

Until now, blocking AI bots was crude: you either let them in or you did not. The new split makes far more sense, because it separates things that affect your business in completely different ways.

Three uses, three decisions

Search

brings traffic

Bots that index content in order to answer queries later. The classic search engine crawlers.

Agent

can earn citations

Bots acting live on behalf of a specific user, during their conversation with an assistant.

Training

usually gives nothing

Crawlers that fetch content to absorb it permanently into a model. No referral, no citation.

Each category affects your traffic differently, which is why one catch-all switch was too blunt a tool.
The sensible question is not "let AI in or not", but "what exactly does this bot do with my content, and what do I get in return".

The trap: Googlebot is a mixed bot

The three-category split has one fundamental flaw: a good share of bots do not fit into a single box. Googlebot indexes pages for the search results and, with the same fetch, feeds the AI features in Search. The same goes for Bingbot and Applebot. Cloudflare calls them multi-purpose crawlers and applies a simple rule to them: every behaviour counts, and the most restrictive rule wins.

In practice this means that a crawler doing both Search and Training is treated as a training crawler wherever Training is blocked. So if you have training blocked and your site shows ads, after 15 September you may cut off the bot that brings you organic traffic.

Most exposed are those who once ticked the catch-all option to block AI bots. Back then it was a single decision; today its meaning is changing. The setting stays the same but starts working differently, and without any action on your part.

To be fair: the sources disagree here

Some reports on the announcement claim that the problem does not concern Googlebot, because Google provides a separate token, Google-Extended, for opting out of training without losing your place in the results. That is true, but it concerns control on the robots.txt side. Cloudflare classifies bots by their actual behaviour on your server and explicitly lists Googlebot among multi-purpose crawlers. So treat this as a risk to check on your own site, not as a settled fact in either direction.

Who the new defaults will cover

The change in default values will cover three groups:

  • New Cloudflare customers: everything connected after the deadline.
  • New domains of existing customers: the old ones keep what they have, the new ones start with the new defaults.
  • All existing accounts on the free plan: and this is the group that, in Poland, accounts for the most sites.

If you are on a paid plan with your own rules, nobody will change them for you. But, and this is the crux, the logic of how they are interpreted is changing, so an identical setting may produce a different effect after 15 September than it does today. Fifteen minutes in the dashboard saves a month of wondering why traffic has dropped.

The biggest risk is not in the new policy but in the old setting everyone has forgotten about.

Why Cloudflare is doing this

The reason is summed up by one statement from Matthew Prince, the head of Cloudflare: since most traffic on the internet no longer comes from people, it is necessary to move faster and go further so that a sustainable ecosystem can emerge. There are concrete numbers behind that statement.

In June 2026 crawlers associated with model training accounted for 50.6% of all traffic, while search engine bots accounted for 10.7%. It shows even more sharply in the ratio of fetches to referred users: Anthropic's crawler fetched 38,000 pages for every one referral to a publisher, and OpenAI's made 1,091 fetches per referral.

Fetches per one referred user

Anthropic crawler 38,000 : 1
OpenAI crawler 1,091 : 1

50.6%

of traffic: training crawlers (June 2026)

10.7%

of traffic: search engine bots (June 2026)

The bars show the scale of the imbalance: next to Anthropic the OpenAI bar is barely visible, although 1,091 fetches per referral is a lot too.

You can argue about whether it should be Cloudflare deciding who gets access to other people's content. But the diagnosis itself is hard to dispute: the deal the open internet stood on (I give content, the search engine gives me a user) has simply stopped applying in the AI layer. Regulators are trying to solve a similar problem from the other side, which I wrote about when covering the decision of the UK's CMA on AI Overviews.

From Pay Per Crawl to Pay Per Use

The second part of the announcement is about money. Cloudflare is turning last year's Pay Per Crawl idea, a toll gate where the publisher charged for the fetch of a page itself, into a Pay Per Use model. The change is logical: payment happens when the content actually feeds a result or an answer, not when a bot simply downloads it.

Two technical elements come with it: an attribution dashboard showing how content is being used, and a monetisation gateway based on the x402 micropayment protocol, which uses the HTTP 402 response code ("Payment Required", a status that sat in the specification for a quarter of a century as reserved for future use).

The launch partners are Ceramic.ai and You.com. Participating publishers get reporting that looks familiar to an SEO: the specific queries for which the content appeared in results, the exact URL and passage, and the average position.

Two partners are not yet a market. But this is the first time "pay for content used in an answer" has a ready mechanism on the technical side, and not just a demand.

For a typical Polish service business website, Pay Per Use will not be a source of revenue and is not worth the time. What is worth remembering is the direction: the infrastructure for measuring and settling what models do with other people's content is growing. In a year or two, reporting from systems like this may be a normal part of analytics.

What to do before 15 September

A concrete list. It takes a quarter of an hour and protects you from a drop that later takes a long time to diagnose:

  1. 01 Check whether you are on Cloudflare at all. It sounds trivial, but plenty of businesses do not know, because the site was often set up by an agency or a previous contractor. The simplest way: the domain's DNS records or the server's response headers.
  2. 02 Find the setting in the dashboard. The application security section, settings, filter by bot traffic, the "Block AI bots" item. Check which state it is in: blocking only on hostnames with ads, blocking everywhere, or switched off.
  3. 03 Move to categories instead of the catch-all switch. Set Search, Agent and Training separately. A sensible default arrangement for most business websites: you allow Search, you allow Agent, and you block Training if protecting your content matters to you.
  4. 04 Take a baseline. Export the crawl stats for the last 90 days from Search Console, take a screenshot of your Cloudflare settings and save a copy of robots.txt. Without a reference point you will not be able to prove to yourself later whether anything changed.
  5. 05 Put a check in your calendar for the end of September. After the deadline, look at the crawl stats and server logs for a drop in Googlebot activity while the content is unchanged. That is the cleanest signal that the block has worked more broadly than you wanted.

Block or allow: how to decide

The decision looks different depending on what the site lives on. Three typical situations:

  • A company or service business website: you live on someone finding you and calling. Blocking here is shooting yourself in the foot: you want to be both in the results and in assistants' answers. Allow Search and Agent, and cut off Training if you wish.
  • A site that lives on ads: every fetch without a referral is a real loss. Cloudflare's defaults are written precisely for you and are probably worth keeping, as long as you keep an eye on the Search category.
  • Unique, costly content: your own research, data, premium material. Here blocking training makes the most sense, and Pay Per Use is worth watching.

The common denominator: separate the decision about being visible from the decision about feeding models. Until recently the two were fused into one; now they can be made separately, and it would be a waste not to use that. How to compete for a place in AI answers in the first place is something I set out in the guide to citations in AI Overviews.

The worst possible decision is no decision. The defaults were written for publishers with ads, not for your site.

FAQ: common questions

Will Cloudflare block Googlebot on my site on 15 September?

Not automatically and not for everyone, but the risk exists and is worth checking. Cloudflare classifies crawlers by three uses: Search, Agent and Training. Multi-purpose bots, among which it counts Googlebot, Bingbot and Applebot, are judged on all of their behaviours, and the most restrictive rule applies. So if you have blocking of the Training category switched on, a bot that combines indexing with feeding AI features may be treated as a training crawler and blocked on pages with ads. Check the settings in the dashboard before the deadline.

Who will the new defaults cover?

The new default values will cover new Cloudflare customers, new domains added by existing customers and all existing accounts on the free plan. If you are on a paid plan with your own rules configured, your settings will not change by themselves, but the logic by which they are interpreted is changing, so they are worth a look anyway. You can make changes in the dashboard at any time before 15 September 2026.

Where in the Cloudflare dashboard do I check these settings?

In the Cloudflare dashboard go to the application security section (Security → Settings) and filter the list by bot traffic. You are looking for the "Block AI bots" item. It can be in one of three states: blocking only on hostnames with ads, blocking on all pages, or switched off. Since 1 July 2026 there are also new presets for the Search, Agent and Training categories, and these give precise control in place of one catch-all switch.

How do the Search, Agent and Training categories differ?

Search covers bots that index content in order to answer queries later: the classic search engine crawlers. Agent covers bots acting in real time on behalf of a specific person, for example an assistant fetching a page during a conversation with a user. Training covers crawlers that fetch content in order to absorb it permanently into a model. The split makes sense because each of these activities affects your business differently: Search brings traffic, Agent can be a source of citations, and Training usually gives nothing in return.

Will blocking AI crawlers hurt my visibility?

It can, if you block too broadly. Cutting off the Training category does not in itself remove you from search results, but cutting off search engine bots or agents that fetch content live can cut you out of citations in AI answers, which is the place that increasingly captures the user's attention. That is why, instead of one catch-all block, it is better to set the categories separately and decide deliberately what you let in.

What does Pay Per Use offer and is it worth looking into?

Pay Per Use is the successor to the Pay Per Crawl model. The difference matters: previously a publisher could charge for the fetch of a page itself, now payment happens when the content actually feeds an answer or a result. For this Cloudflare has launched an attribution dashboard and a monetisation gateway based on the x402 micropayment protocol, which uses the HTTP 402 code. The launch partners are Ceramic.ai and You.com. For most Polish sites this is a curiosity and a sign of the direction for now, not a source of revenue, but the direction is worth knowing.

Worth remembering

  • From 15 September 2026, by default: Search allowed, Agent and Training blocked, but only on pages that show ads.
  • The new defaults will cover new customers, new domains of existing customers and all accounts on the free plan.
  • Multi-purpose bots (Googlebot, Bingbot, Applebot) are judged on all of their behaviours, and the most restrictive rule wins.
  • The biggest risk is for those who once clicked "Block AI bots" and forgot about it: what that switch does is changing without any action on their part.
  • Take a baseline before the deadline: an export of crawl stats from Search Console, a screenshot of your Cloudflare settings and a copy of robots.txt.
  • Pay Per Use pays for content being used in an answer, not for the fetch itself, but for now it runs with two partners, so treat it as a sign of the direction.

Summary

The change itself is reasonable: separating bots by what they do with content is a step in the right direction and something the industry has been asking for for a long time. The risk lies elsewhere: in old settings that suddenly start to mean something different, and in default values written for publishers, not for the average business website.

A quarter of an hour in the dashboard before 15 September is cheaper than two months of looking for the cause of a traffic drop.

Do three things: check whether you are on Cloudflare, look at the bot settings and record the state of crawling before the deadline. The rest is a deliberate decision you make yourself, instead of receiving it as a gift from the default settings.

Not sure how your site is set up?

I will check the bot configuration, robots.txt and crawl stats, then tell you plainly what to change before the deadline. No scaremongering and no year-long package.

Book an SEO consultation

How this text was written

I will not pretend it was written entirely by hand, by candlelight. For the research, the proofreading and the code of this page I used a language model. It is a good tool for digging through documentation and catching the typos you no longer see on the third read of your own text. The English version was also prepared with a model, from my Polish original.

What the model did not do: it did not choose the topic, form the arguments or decide what this data means for your site. That stays with me, as does the last word on what goes into the text and what comes out of it.

Where the model helped

  • digging through sources and organising data
  • proofreading and the English version of the text
  • code, page layout and infographics

What stays with me

  • the choice of topic and what is worth saying about it
  • the arguments, the reading of the data and the conclusions
  • fact-checking and the decision on what goes into the text
Sebastian Baryś – freelance SEO specialist

Written by

Sebastian Baryś

Freelance SEO specialist · Częstochowa, Poland, and remote

I started when nobody had dreamt of Core Web Vitals and rankings were checked by hand. Today I work on making companies visible not only in Google, but wherever people ask questions. I write only about things I check myself in client projects.

SEOGoogle AdsTechnical SEOSEO in the AI era

Want to be sure that Google can still see you?

I will check your crawler configuration, indexing and visibility, and you will get recommendations in plain language, with the reasoning behind them, not from a generator.

Get in touch