Cloudflare aims to eradicate AI scraper bots for good
Cloudflare has announced a new feature that will allow web hosting customers to prevent AI bots from scraping their website content without permission.
Companies responsible for AI tools, particularly those involved in training, have come under fire in recent months for their use of content published online.
According to Cloudflare, the new tool is a response to widespread dissatisfaction among website owners and publishers over AI bots that collect data, and is also intended to protect content creators on the internet.
In his announcement, Cloudflare wrote: “We’re hearing from customers that they don’t want AI bots visiting their sites, especially those that do so dishonestly. To help, we’ve added a brand new one-click feature to block all AI bots.”
The tool is now available to all customers, including those on the free version of Cloudflare.
Previously, website owners could use the robots.txt file to block bots from crawling their sites, but with varying degrees of success. The new tools provide a more robust defense against content scraping by using machine learning to identify and block bots.
Cloudflare’s system uses digital fingerprinting to distinguish bots from legitimate users.
The company said the enhanced protections are in response to significant AI bot activity on its network. Bytespider, GPTBot and ClaudeBot were among the most commonly observed, but despite earlier blocking attempts, AI bots were able to access more than a third (39%) of the top one million sites served by Cloudflare in June.
Fortunately, the new feature is available to all Cloudflare customers, including those on the free tier. By navigating to the Security > Bots menu and blocking âAI Scrapers and Crawlers,â users can protect their website from unwanted AI bot activity.