Should your small blog or Substack charge AI crawlers, or just block them?
Updated August 6, 2026
Charge AI crawlers only if you already get enough traffic that a few cents a fetch could add up to real money. Otherwise block by default and stop thinking about it. That's the whole decision in one line, and the numbers back it up: Cloudflare's pay-per-crawl floor sits at $0.01 per fetch, and for a personal WordPress blog or a small Substack, that price tells you almost everything you need to know about whether charging AI crawlers is a real revenue plan or just a way to say no with slightly less friction.
What actually changes on September 15
Cloudflare has said that starting September 15, 2026, newly onboarded domains get a new default: training and agent crawlers are blocked automatically on pages that carry ads, while search crawlers like Googlebot stay allowed so the site keeps showing up in normal search results. You can override any of it per bot. Allow a crawler for free, block it outright, or charge it under pay-per-crawl, which runs on the HTTP 402 "payment required" status code, a part of the web standard that sat unused for thirty years until this became its actual job. The mechanics of setting that up, including the x402 payment standard behind the charging option, are covered in more detail on our page on getting paid when AI reads your site.
How much money is actually on the table
Be plain with yourself here: for a blog with a few hundred or a few thousand visits a month, income from charging AI crawlers is likely to be small, and it may be nothing. You set your own price, but demand depends on how many AI companies bother to crawl a site your size at all, and most small sites simply don't get crawled often enough for a per-request fee to add up to real money. A site pulling in tens of thousands of crawl requests a month might see a few dollars to a few tens of dollars; most personal blogs won't get anywhere near that volume. Treat it the way you'd treat Meta's USDC payouts to creators, which are real but still rolling out market by market and won't reach most small publishers this year either. Pay-per-crawl is a marginal option sitting next to your actual income, not a replacement for it. If direct reader support is what you're after, a non-custodial tip jar where readers pay you straight to a wallet you control is a more dependable bet than waiting on bot traffic to pay rent.
The traffic you'd be trading away
Here's the part people skip. Blocking AI crawlers feels like it should cost you visibility, but recent referral data doesn't back that up yet. In July 2026, Google was still sending roughly 88% of search referral traffic across the web, while ChatGPT's share, even after a year of growth, sat under 1%. So today, blocking AI crawlers costs most small sites very little direct traffic, because AI chatbots aren't sending much traffic to begin with. The flip side matters too: if you're hoping to get "cited" by an AI answer engine and have that turn into real readers later, blocking now closes a door that hasn't really opened yet anyway. Neither choice is obviously wrong. It just means you shouldn't block out of fear of losing traffic you probably weren't getting, and you shouldn't allow out of hope for traffic that hasn't shown up yet either.
What Substack and WordPress owners can actually do
Self-hosted WordPress gives you the most control. Robots.txt lets you name individual bots like GPTBot, ClaudeBot, or PerplexityBot directly, and if the site sits behind Cloudflare, its AI Crawl Control panel reads a similar bot list and lets you set allow, block, or charge per crawler without touching code. Hosted platforms like Substack give you less to work with, since you don't manage the server, but the model is spreading past WordPress and Cloudflare's own customers: Cloudflare announced AI crawl controls built for beehiiv, a newsletter platform, in June 2026, which suggests hosted publishing tools are being pulled into this whether their writers ask for it or not. If you're on a platform you don't control, check its settings before assuming you have no options at all.
| Setting | What it does | Realistic outcome for a small blog |
|---|---|---|
| Allow all crawlers | AI bots read everything free, same as search engines | Best odds of eventual AI citations, zero direct income |
| Block all crawlers | Training and agent bots get refused; humans unaffected | Protects your text, costs little traffic today given AI's sub-1% referral share |
| Charge via pay-per-crawl | Bots pay your set price (minimum $0.01) or get blocked | Cents to low dollars a month for most small sites; more only at real scale |
| Mixed, per bot | Allow search engines, block or charge training and agent bots | What most small sites land on once they've thought it through |
The simplest default if you don't want to think about it
If you'd rather not spend an afternoon reading crawler documentation, the safe middle setting is this: keep search engines allowed so people can still find you, and block AI training and agent crawlers by default, the way Cloudflare's own new default will do for you automatically after September 15 if you don't touch anything. Skip setting up charging unless you already get enough traffic that a few cents a fetch might add up to a few dollars a month, because the setup time isn't worth it below that. You can always turn charging on later for a specific bot that keeps hammering your server, once you can see in your logs that it's worth the trouble.
Don't let a bot read for free what you wouldn't let a stranger copy and republish for free, and don't expect pay-per-crawl to replace real income anytime soon. For a support option that pays today instead of after September 15, see the tip jar generator page and get a wallet-based tip jar running in less time than it'd take to read through Cloudflare's bot list.