Cloudflare has introduced Pay Per Crawl to allow content owners to charge AI crawlers for content access.
Pay Per Crawl uses HTTP 402 responses and integrates with existing web infrastructure for payment-based access.
Publishers can set domain-wide flat pricing and choose to allow, charge, or block specific crawlers.
AI crawlers must authenticate using Web Bot Auth with Ed25519 key pairs and HTTP message signatures.
Crawler requests can follow reactive (discovery-first) or proactive (intent-first) payment flows.
Cloudflare acts as the Merchant of Record, handling billing, aggregating events, and distributing earnings.
Pay Per Crawl is in private beta and aims to enable future programmatic content licensing and agent-based negotiations.
Get notified when new stories are published for "🇺🇸 Hacker News English"