Last week, I found myself staring at a dead-links report for an old client project. Over 300 pages. I needed to know which URLs were returning 404s, which ones were silently redirecting, and which ones had just given up entirely.
My first instinct was to write a quick loop in Node.js using fetch .
I hit two massive walls immediately:
The Rate Limit / IP Ban wall: After about 15 requests, the target servers started dropping my connection.
The Timeout wall: My script hung for 5 minutes because three random URLs just never responded.
If you've ever tried to bulk check HTTP statuses at scale, you know the pain.…