Cloudflare launched Pay Per Crawl, allowing content owners to charge AI crawlers for access.
Content owners can choose to allow, charge, or block crawlers using HTTP status codes.
Pay Per Crawl uses HTTP 402 Payment Required and standard authentication to handle pricing.
Publishers set a flat per-request price and can grant exceptions for specific crawlers.
Crawlers must use Web Bot Auth with signed requests to verify their identity.
Two payment flows exist: reactive (402 discovery then payment) and proactive (max-price header).
Cloudflare handles billing and settlements between crawlers and publishers.
Future potential includes dynamic pricing, granular licensing, and agent-based negotiation.
Get notified when new stories are published for "General AI News"