I stopped double-scraping jobs with a Postgres lease table.

I made cron runs idempotent with a single SQL function.

I got rid of “why is this job missing?” by logging per-source runs.

Works on Next.js 14 + Supabase + Vercel cron.

Context

I’m building a job board for Psychiatric Mental Health Nurse Practitioners.

It scrapes 200+ jobs daily. Multiple sources. Different HTML. Different APIs. The usual pain.

At ~8,000 active listings and ~2,000 companies, one bad cron day hurts. Duplicates show up. Or worse — jobs silently don’t.

My first version was “just run scraping in a cron route.” Brutal.

Vercel cron can overlap. Deploys can overlap. And if a source gets slow, the next run starts anyway. I spent 4 hours “fixing dedupe” and most of it was wrong because the real issue was concurrency.

So I moved coordination into Postgres. Not Redis. Not queues. Just SQL.

1) I stopped trusting cron timing.…