Every so often a directory like mine stops adding tools and starts shipping broken links instead. I run one: about 4,323 AI tool listings. I had been assuming "the crawler refreshes the descriptions" was close enough to "the links still work". So I spent a morning checking whether the stored URLs actually load. The method was simple, and it collapsed on the first control group.
What I ran
Sample: 300 of 4,323 listings (6.9%) — the top 150 by the traffic number I display, plus 150 random from the rest. One machine, one pass, Node fetch , concurrency 16, follow redirects, 15-second timeout.
outcome count
2xx / 3xx 193
4xx 21
5xx 2
no TCP or TLS answer at all 84
First read of that table: "28% of my catalog is dead." Wrong, and I can prove it's wrong.
The control group that killed the method
Two of the 84 were notion.so and writesonic.com . Both are undeniably alive companies.…