Summary & Key Takeaways ​

• The article discusses the significant resource consumption by abusive web crawlers.
• git.kernel.org reports that scrapers use more CPU than all legitimate access combined.
• Simon Willison expresses concern for his own project, Datasette, which serves many crawlable pages.
• The issue highlights the "background radiation" of unwanted automated traffic.
• It raises questions about the operational burden and ethical implications of AI-driven crawling.

Our Commentary ​

This is genuinely unsettling. The idea that AI agents are just churning away, consuming more resources than actual human or legitimate programmatic access, feels like a slow-motion denial-of-service attack. We've always dealt with bots, but the scale and potential for abuse with AI are different. I wonder how many other services are silently struggling with this. It's a stark reminder that "free" access isn't free for the infrastructure providers.