Menu

A job postings API over 18,000 company career sites, measured: one night's crawl, its cost and one search
📰

A job postings API over 18,000 company career sites, measured: one night's crawl, its cost and one search

DEV Community: datascience·Adderley Data·9 days ago
#41NwvLPa
#devtodatascience#science#devto#dev#postings
Reading 0:00
15s threshold

Most companies publish their open roles through an applicant tracking system (ATS), and the ATS serves each company's job board through a public API or feed so that the company's own careers page can show it. We read about 18,000 of those boards on eight systems, seven of the systems every night and Workday every second night, and keep the postings in one index that a single API call searches. This is the index as it stood at 16:26 UTC on 7 November 2026, what one night of reading it took, and one search run against it from start to finish. Every figure comes from the runs' own records; times and dates are UTC.

Where the boards come from

No ATS vendor publishes its customer list. Common Crawl's URL index is the next best thing: it records every page Common Crawl's crawler captured, keyed in SURT form (the host reversed, then the path), so the boards of one system sit together in one sorted run of keys.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.