Cloudflare wants customers to be able to block specific AI-bots
The web traffic specialist is coming up with plans to allow customers to block AI-bots that come to 'scrape' their content. An online marketplace must also give sites the ability to make AI-makers pay for that content.
Cloudflare wants to launch an online marketplace next year where website owners can sell access to their site to AI-bots. In the first phase, Cloudflare-customers will get a button that allows them to block all AI-bots.
This is specifically about bots that come to 'scrape' or download the content of a website so that texts, images, and videos can be used to train AI or summarize the content of a site. The AI-models cost the sites clicks and income because users no longer click through to the original source. "If you don't compensate the makers in some way, they will stop making new content, and we need to find a way to solve that," says Matthew Price, CEO of Cloudflare, to tech site TechCrunch.
Anti-robot protocol
Web scraping is technically the indexing of the internet by crawler bots. The process is used by various search engines, and via the so-called Robots Exclusion Protocol, a website can indicate in its code which pages crawlers are not allowed to visit and index. However, this protocol is not mandatory, and not every new startup adheres to it. This summer, tech magazine Wired accused AI-startup Perplexity of bypassing the protocol and plagiarizing text that was specifically excluded.
Cloudflare, which manages traffic for many websites, now has a more binding solution. The company has been giving its customers a free button over the past weeks that allows them to block all AI-bots in one click. Price also announces a new free tool called AI Audit, with which website owners can get an overview of which AI-models are accessing their content, when, and how.
Once everything is rolled out, it must be possible in the future to allow specific AI-bots for compensation. Companies that have struck deals with providers like OpenAI (such as Time magazine) would only allow bots from that company and not others. The details of this plan and what it will look like for smaller publishers and websites are still unclear. However, the block button seems to be a first step.