In mid-October I looked at Cloudflare analytics for hipa.ai , the clinical-trials search site I build alone. On a typical day there were about 455,000 requests marked DYNAMIC and another 141,000 marked MISS . Every one of them went through to my Next.js origin on Azure App Service, which rendered the page from scratch against MongoDB and Elasticsearch.
Very few of those requests were people. When I later reviewed our own client-side analytics beacon, about 95% of the events came from bots too. Real people were somewhere around 250–370 visitors a week.
This post covers who was sending the traffic, the cache rule that took it off the origin, and the four ways that rule could have served wrong pages for a full day.
Who was hitting the origin
There were three groups:
A headless residential-proxy botnet. User agent X11; Linux x86_64 … Chrome/154 , about 8,500 distinct US IPs , ran JavaScript, ignored robots.txt. It looped over / , /platform , /for-* and /auth/* all day.…