I run an AI tools directory with about 1,700 tools. Visitors want two things from it: which tools are actually popular , and which ones are taking off right now . For a while I scraped SimilarWeb for that. Then SimilarWeb started blocking my server's IP, and paying for their API wasn't an option for a side project.
It turns out you can get surprisingly far with two free, public lists. This post shows the whole monthly job — about 100 lines of Node — and the traps I hit.
The two data sources
Tranco top 1M ( tranco-list.eu ) is a research ranking of the top million domains. It combines several popularity sources and averages them over 30 days, which makes it far harder to game than any single list. You get an exact rank ( 1,openai.com -style CSV in a zip), updated daily.
CrUX top lists (the zakird/crux-top-lists repo on GitHub) are derived from the Chrome UX Report. They don't give exact ranks, only buckets: top 1k, 5k, 10k, 50k, 100k, 500k, 1M.…