I work at UN. UN job hunting is a particular kind of suffering: openings are scattered across careers.un.org , reliefweb.int , agency-specific portals, and aggregators of varying quality. None of them know my profile. None let me say "only show me P-2 and P-3 protection roles in family duty stations outside Türkiye." So I built one.

This is the story of how I built that — and what the build itself taught me about respecting the tools you scrape from.

The bad idea I started with

My first instinct was simple. Open unjobs.org , it lists everything in one place, write a requests + BeautifulSoup scraper, done. Maybe a Cloudflare Worker, maybe a small Next.js dashboard, ship by Sunday.

I asked Claude to help me scope it. The first thing it did was fetch the site to see the structure.

ROBOTS_DISALLOWED: URL is disallowed by robots.txt rules

Enter fullscreen mode

Exit fullscreen mode

Right. unjobs.org explicitly forbids scraping.…