Cloudflare, which spent the past year introducing tools to help content providers prevent unwanted AI scraping, is launching a marketplace that lets websites charge for the privilege of using a “pay-per-crawl” model. The Internet infrastructure and security company says it is the first to enable blocking AI crawlers by default, providing access only with permission and, if wanted, compensation. As of July 1, AI companies can use Cloudflare’s marketplace to “clearly state their purpose — if their crawlers are used for training, inference, or search — to help website owners decide which crawlers to allow.”

“If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone — creators, consumers, tomorrow’s AI founders, and the future of the web itself,” Cloudflare co-founder and CEO Matthew Prince said in an announcement.

An estimated 20 percent of the world’s websites contract to use a Cloudflare service. Starting this week, they “will be asked if they want to allow AI crawlers, effectively giving them the ability to prevent bots from scraping data from their websites,” writes CNBC. “The company will also allow publishers to charge AI crawlers for access” using a “pay-per-crawl” approach. Cloudflare says pay-per-crawl lays the groundwork for “a new business model.”

TechCrunch reports “Cloudflare is launching the ‘experiment’ in private beta” with a sign-up page for those who would like early access.

“Website owners can choose to let AI crawlers scrape their site for free, or block them altogether,” writes TechCrunch, noting that “Cloudflare claims its tools will let website owners see whether crawlers are scraping their site for AI training data, to appear in AI search responses, or for other purposes.”

“Whereas the Internet previously rewarded creators by directing users to original websites, according to Cloudflare, today AI crawlers are breaking that model by collecting text, articles and images to generate responses to queries in a way that users don’t need to visit the original source,” CNBC says of a shift impacted by chatbots and AI search.

“Waiting for legal or government interventions to remedy the current challenges in our ecosystem isn’t a strategy” but sustainable models that compensate content creators fairly is, says Penske Media Chief Strategy Officer Craig Perreault in one of nearly 40 testimonials Cloudflare provides from notable clients.