Steam Inventory Value prices a Steam inventory from a cache of market prices. A crawler inside Postgres keeps that cache fresh: every minute it may fetch two pages of a game's market catalogue, 100 items a page. The budget is small on purpose, because Steam rations requests.
Two pages a minute has to be shared between every game we support. This post is about how we shared it, why that quietly failed for new games, and the small rule that fixed it.
The rotation
Each game has a refresh interval: 6 hours for Dota 2 and CS2, 12 for TF2 and Rust, 24 for Steam trading cards. The crawler picks the game that is most overdue relative to its own interval:
select appid from crawl_apps where enabled order by ( cycle_started_at is null ) desc , extract ( epoch from now ( ) - cycle_started_at ) / ( refresh_hours * 3600 ) desc nulls first , sort limit 1
Enter fullscreen mode
Exit fullscreen mode
A game that started its round 12 hours ago with a 6-hour interval scores 2.0.…