RSL, blocking, or charging: which should a small site pick
Updated August 6, 2026
More than 1,500 publishers, including Reddit, Yahoo, Medium, and Quora, had already adopted Really Simple Licensing by the time its 1.0 specification became an official industry standard on December 10, 2025. That number tells you RSL has real momentum behind it. It does not tell you that a single AI company is obligated to pay any of those sites a cent.
If you run a small site, you're about to have three doors to choose from, not two. Cloudflare's default for ad-supported pages flips from allowing AI crawlers to blocking mixed-use ones on September 15, 2026, as we covered in our piece on getting paid when AI reads your site. RSL adds a third option that sits between "let them in for free" and "keep them out": publish licensing terms and hope, or arrange, for someone to honor them.
what each option actually does when a crawler shows up
These aren't three flavors of the same mechanism. They're different layers of the stack, and mixing them up is where most site owners get confused.
| Approach | What it actually does | Who enforces it | Money you might see |
|---|---|---|---|
| Block outright | Firewall rule or robots.txt disallow stops the request before it reaches your content | Cloudflare's edge, or your own server config | None. That's the point. |
| RSL licensing terms | Machine-readable file (an extension of robots.txt) states your price and terms: attribution, pay-per-crawl, pay-per-inference | Voluntary compliance by the crawler, unless you're on Fastly's CDN, which checks for a license before letting a declared crawler through | None guaranteed. Depends entirely on the AI company choosing to honor it. |
| Cloudflare pay-per-crawl / pay-per-use | Server returns HTTP 402 with a price; a crawler that wants in pays through the x402 rail to complete the fetch | Cloudflare acts as merchant of record and blocks anyone who doesn't pay | Actual stablecoin, though amounts run small, often fractions of a cent per fetch |
RSL is a licensing signal, not a cash register
Be honest with yourself about what RSL does. It gives you a standardized, machine-readable way to say "you may train on this for a fee" or "indexing yes, AI training no," embedded right in robots.txt. That's genuinely useful: it replaces a patchwork of ad hoc "please don't scrape me" blog posts with one format that Yahoo, Reddit, Medium, Quora, Ziff Davis, and now over a thousand other publishers all speak.
What it doesn't do, by itself, is collect money or block anyone. RSL has no built-in payment rail and no enforcement layer of its own. The RSL Collective's partnership with Fastly gives publishers on that specific CDN a way to check whether a declared crawler has agreed to a license before serving it a page, but if you're not on Fastly, RSL terms are a polite request sitting in a text file. Whether OpenAI, Anthropic, Google, or anyone else respects it is up to them, not up to the standard.
blocking is the new default, and it's the bluntest tool
From September 15, 2026, Cloudflare blocks mixed-use crawlers (bots that can't or won't declare whether they're indexing for search or training a model) by default on any ad-supported page, for new domains and free-tier accounts that haven't touched the setting. That's a meaningful shift from "AI crawlers are welcome unless you say otherwise" to the reverse.
Blocking is simple and it costs nothing to set up. It also earns you nothing. If your actual goal is compensation rather than exclusion, an outright block is a starting position, not an end state. Plenty of small site owners will land there anyway, because it's the path of least effort and because a blank "no" is easier to reason about than negotiating terms with a bot.
charging is where the actual payment rail lives
Cloudflare's pay-per-crawl product, now being folded into a broader "pay-per-use" model, is the one option here that moves real money. A publisher sets a price per crawler, Cloudflare returns an HTTP 402 status with that price attached, and a crawler willing to pay settles the request through x402, the payment standard the Linux Foundation stood up as its own foundation on July 14, 2026 with Visa, Mastercard, American Express, and Stripe among the paying members, settling in stablecoins over ordinary web requests. Cloudflare is also piloting deals with search and answer engines like You.com so publishers get paid when their content shows up inside an answer, not only when a crawler fetches it.
This is the same rail, conceptually, as a creator posting a wallet address for tips through something like a non-custodial tip jar generator, just automated and running server-to-server instead of person-to-person. It's part of a broader shift toward creators and publishers getting paid directly rather than waiting on ad revenue, the same shift that has Meta paying some creators in USDC since April 2026, as we detail in our piece on Meta's USDC payouts.
what a small site should probably actually do
Most small sites don't need to pick exactly one. A reasonable stack looks like: publish RSL terms so your preferences are on record in a format the industry is converging on, turn on Cloudflare's blocking for anything that won't identify itself, and enable pay-per-crawl or pay-per-use for the specific crawlers Cloudflare already categorizes and that are willing to pay. That gets you a documented position, a default that protects you from silent scraping, and an actual payment mechanism for the crawlers that respect it, layered rather than chosen from a menu.
The honest limitation is this: even doing all three, most small sites are going to see very little money. Pay-per-crawl amounts run to fractions of a cent per fetch, RSL has no enforcement outside a single CDN partnership, and the AI companies with the largest budgets can often get comparable content from a mirror, an archive, or a competitor that never turned any of this on. None of these three tools force payment out of anyone who doesn't want to pay.