Cloudflare introduced an accountable operator designation system to apply to AI crawling. It also released a feature called "Disallow AI Training" that refuses AI training while keeping search exposure, it said on Sept. 18.

Cloudflare previously said website owners should directly control how AI uses their content, and it presented conditions for crawler operators to gain ongoing access permission. With this announcement, Apple, Google and Microsoft were designated as accountable operators. The three companies met the criteria or presented a specific schedule to meet them.

Crawlers other than Apple, Google and Microsoft will have access blocked if site owners set "block training."

According to the company, mixed-use crawlers that handle both search and AI training account for 36.6 percent of Cloudflare network crawler traffic. Fewer than 1 percent of sites block search crawlers, but 17 percent restrict AI training.

Matthew Prince (매튜 프린스), Cloudflare co-founder and CEO, said, "The way to make the internet a better place is to allow the people and companies that create the web to have real control over how their work is used," adding, "We will continue to work with these companies."

Cloudflare said those designated as accountable operators must provide site owners with a way to refuse AI summaries. For now, site owners must set this separately for each operator. Cloudflare said it plans to make it possible to manage this in one place by early next year.