There’s a pretty significant difference in request rate. A tool trying to search and summarize will hit a search engine once, and each website maybe 5 times (if every search engine link points to the site).
A bot trying to scrape content from a website can generate thousands or tens of thousands of requests per second.
It isn’t opt in.
You can block all bot page scraping, and also block user initiated AI tools or you can block no traffic.
There isn’t an option to block bot page scraping but allow user initiated AI tools.
Because, as the article points out, Cloudflare is not able to distinguish between the two
There’s no appreciable difference on how they affect systems between the two for site owners.
There’s a pretty significant difference in request rate. A tool trying to search and summarize will hit a search engine once, and each website maybe 5 times (if every search engine link points to the site).
A bot trying to scrape content from a website can generate thousands or tens of thousands of requests per second.