Cloudflare's September 15 AI crawler change: what it actually did to your site

Updated September 18, 2026

On September 15, 2026, Cloudflare added a setting called Disallow AI Training: it writes a no-training preference into your robots.txt, keeps accountable crawlers like Googlebot allowed for search, and blocks every other training crawler. If your site already existed, Cloudflare says your settings carried over and you do not have to do anything. New domains now pick one of two presets, depending on whether the site runs ads.

What actually changed on September 15

For most of the summer the story was "Cloudflare will start blocking AI bots on September 15." That was never quite right, and the version that shipped is softer and more useful than the headline. Cloudflare's post from that day, "Have it both ways: stay discoverable in search while disallowing AI training", did three things.

• It added a fourth option for the Training category, Disallow AI Training, next to Allow, Block on pages with ads, and Block.
• It named seven crawler operators it treats as "accountable": Apple, Google, Microsoft, Amazon, Anthropic, Meta and OpenAI.
• It set two presets for domains added from that date, one for sites that earn money from advertising and one for sites that do not.

The groundwork was laid on July 1, 2026, when Cloudflare's "Your site, your rules" post split AI traffic into three categories on every plan, Free included. Search is a crawler that "collects or indexes your content, so it can answer questions about it later." Agent is automated behavior "acting, usually in real time, on a person's behalf." Training is a crawler "taking your content to train or fine-tune a model." Our pay per crawl setup guide walks through where those controls sit.

Why mixed-use crawlers were the problem

The three categories had a hole in them. Googlebot, Applebot and Bingbot each fetch a page once and use it for more than one thing. Under the July rules Cloudflare was plain about the consequence: a multi-purpose crawler is allowed or blocked "according to all of their behaviors." Block Training, and Googlebot got blocked too, search and all.

For a small site that is a terrible trade. Very few site owners want to vanish from Google to make a point about model training. Cloudflare's own numbers show how lopsided the preference is: fewer than 1% of Cloudflare sites choose to block Search bots, while 17% turn on some mechanism to block training. People wanted "index me, don't train on me," and the old switch could not say that.

What Disallow AI Training does

Cloudflare describes the new setting in one sentence: Bot Preference Sync "publishes the applicable no-training preference in robots.txt. Accountable mixed-use crawlers remain allowed for search. Every other training crawler is blocked."

In practice that works in three layers.

1. A written preference. Cloudflare adds the right lines to your robots.txt for each operator. For Google that is a disallow for Google-Extended; for Apple it is Applebot-Extended.
2. Search stays open. The accountable crawlers keep reaching your pages so you stay in their search results.
3. Enforcement at the edge. Cloudflare says it can publish the preference, see who is crawling, "and block the ones that ignore it." A training crawler that is not on the accountable list is simply refused.

There is one limit worth knowing. Disallow AI Training exists only for the Training category, not for Search or Agent, and it applies to the whole site. Cloudflare can detect which pages serve ads, but it says that list "is too large and changes too frequently to enumerate in robots.txt." So you cannot say "no training on my ad pages, training fine everywhere else" with this setting. It is all or nothing for training.

Does it hurt search rankings?

The operators say no, and Cloudflare made that a condition of the list. Google's crawler documentation states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Apple's Applebot-Extended page says the token "does not crawl webpages" and that rules for it are not considered in ranking for Search. Microsoft is the exception on mechanics: Bing honors a NOARCHIVE meta tag today and, according to Cloudflare, is building a site-level no-training preference in robots.txt targeted for early 2027.

Who counts as an accountable crawler

To be treated as accountable, an operator has to meet, or commit to meeting, four requirements that Cloudflare lists in the September 15 post:

• a way for site owners to opt out of AI training, through robots.txt or a similar standard;
• a way to opt out of AI summaries, set with the operator directly now and through Cloudflare next year;
• URL-level visibility into which pages were made available for training, plus metrics on how content appeared in search;
• assurance that opting out of training will not affect traditional search results.

Seven operators qualified on launch day: Apple, Google, Microsoft, Amazon, Anthropic, Meta and OpenAI. The list matters because it decides who gets the "search yes, training no" treatment. A crawler outside it that wants your pages for training is blocked under Disallow AI Training, however it identifies itself.

The four settings side by side

Read the last row carefully. If you choose Block for Training because it sounds strongest, a mixed-use crawler like Googlebot is blocked with it. That is the old trap, still available on purpose. For almost every creator site, Disallow AI Training is the setting that does what people meant when they said "block AI."

Did my existing site change?

Probably not, and this is where a lot of summer coverage went wrong. In July, some outlets reported that the September 15 default would also hit every existing site on the Free plan. Cloudflare's own July post described the new defaults as applying to new domains, and its September 15 post answers the question directly. Asked what site owners need to do, it says: "Nothing, in almost every case. Your current settings carry over on their own."

What did happen is a quiet translation of older settings into the new ones. Cloudflare published the mapping:

That last row is the one that surprises people. If you set Training to Block after July to keep AI out, Cloudflare has now moved you to Disallow AI Training, so Googlebot and the other accountable crawlers can reach your pages again for search. For nearly everyone that is the better result. If you truly want those crawlers gone, including from search, you have to go back in and choose Block yourself.

If you added a domain to Cloudflare on or after September 15, you were offered one of two presets. A site that earns money from advertising gets Search Allow, Training Disallow AI Training, Agent Block on pages with ads. A site without ads gets Allow across all three. Cloudflare's reasoning for the ad preset is blunt: "Ad revenue depends on a human actually seeing the page. Training replaces that visit with an answer; agents fetch the page with nobody there to see the ads."

How to check your own settings

This takes about five minutes per domain, and it is worth doing once rather than trusting a migration table.

1. Open the Cloudflare dashboard and select the domain.
2. Go to the domain's Security settings, where Cloudflare says the Search, Training and Agent controls live. The AI Crawl Control section shows which crawlers are actually visiting.
3. Read the three categories and write down what each one is set to.
4. If Training reads Disallow AI Training, open https://yourdomain/robots.txt in a browser and look for the no-training lines Cloudflare publishes, such as Google-Extended.
5. If Training reads Block and you still want search traffic, switch it to Disallow AI Training.

One caveat on detection. Cloudflare's AI Crawl Control documentation says Free plan detection relies on user agent strings, while paid plans get more thorough detection using Bot Management. A crawler that lies about its name is harder to catch on Free. The robots.txt preference still stands, but enforcement is sharper on a paid plan.

If your site is on Substack or another hosted platform that does not run through your own Cloudflare account, none of this applies to you directly. Your platform's own settings decide it. Our guide for small blogs and newsletters covers what those owners can do.

Where Content Signals and pay per use fit

Two related pieces are easy to confuse with the September 15 change.

Content Signals are the lines Cloudflare's managed robots.txt adds to state how content may be used. In July, Cloudflare added a fourth field, use, with three values: reference (the default: "index, excerpt, and link back"), immediate ("interact, but store and reuse nothing") and full ("summarize and reproduce"). Its example line reads Content-Signal: search=yes,ai-train=no,use=reference. Content Signals state a preference. Disallow AI Training is the part that enforces one, by blocking training crawlers that do not honor it.

Getting paid is a separate track again. Cloudflare's docs still list "Charge for crawl" as a private beta, and on July 1 it said it was evolving pay per crawl toward pay per use through experiments with Ceramic.ai, which pays when content appears in search results, and You.com. Cloudflare calls these experiments, and they are not something a small site switches on today. If charging is what you are after, start with RSL, blocking, or charging and the explainer on x402, the payment standard behind crawler payments.

What a small creator site should pick

For a blog, portfolio or newsletter archive, the sensible setup after September 15 is short.

• Search: Allow. This is where readers still come from.
• Training: Disallow AI Training. You stay in Google, Bing and Apple search, you state a no-training preference, and crawlers that ignore it get blocked.
• Agent: Block on pages with ads if you run ads; Allow if you do not. An assistant fetching your page for a reader is closer to a visit than to training, and on a page without ads it costs you nothing you were earning.

Then leave it. The value you get from AI crawlers as a small site is mostly about whether you are found, not whether you are paid. Direct reader support still pays more reliably than bot traffic, which is why a tip jar you control belongs on the page long before a crawler price does. If you do want the fuller background on the pay-or-block question, get paid when AI reads your site sets out the choices.

This describes Cloudflare's settings as published on July 1 and September 15, 2026. Dashboard labels, the accountable list and defaults can change; check your own domain's settings before you rely on this page.