Menu

A fair rotation that starved the newcomers: fixing a round-robin price crawler
📰

A fair rotation that starved the newcomers: fixing a round-robin price crawler

DEV Community: postgres·CC Ong·about 4 hours ago
#Ja1PXFjg
#devtopostgresql#devto#dev#game#first
Long article (1019 words) — no AI summary yet.
Reading 0:00
15s threshold

Steam Inventory Value prices a Steam inventory from a cache of market prices. A crawler inside Postgres keeps that cache fresh: every minute it may fetch two pages of a game's market catalogue, 100 items a page. The budget is small on purpose, because Steam rations requests.

Two pages a minute has to be shared between every game we support. This post is about how we shared it, why that quietly failed for new games, and the small rule that fixed it.

The rotation

Each game has a refresh interval: 6 hours for Dota 2 and CS2, 12 for TF2 and Rust, 24 for Steam trading cards. The crawler picks the game that is most overdue relative to its own interval:

select appid from crawl_apps where enabled order by ( cycle_started_at is null ) desc , extract ( epoch from now ( ) - cycle_started_at ) / ( refresh_hours * 3600 ) desc nulls first , sort limit 1

Enter fullscreen mode

Exit fullscreen mode

A game that started its round 12 hours ago with a 6-hour interval scores 2.0.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.