Artificial intelligence companies, such as OpenAI and Anthropic, are using bots to collect data from websites, which puts significant pressure on the servers of these sites. This practice, known as “web scraping,” involves bots repeatedly accessing pages to gather large amounts of information, which can increase the operational costs of websites by increasing traffic and resource usage.
Just like a swarm of locusts that devours crops, these bots pose a challenge for website administrators, who must deal with the increased demand on their servers, leading to additional costs and performance issues. Additionally, blocking these bots is complicated, as they often disguise themselves as legitimate traffic. Affected companies are beginning to take measures to mitigate the impact, but the situation remains problematic.
There are paid solutions such as Amazon WAF to prevent these bots from overwhelming your website, making it slower or even inaccessible, and also increasing your cloud hosting bill.
If you can’t afford a service like Amazon WAF, you can use the free tier of Cloudflare and from the Security menu -> Bots activate these two options:
This way, Cloudflare will attempt to block all AI bot traffic that wants to indiscriminately crawl your website.