Menu

1 / 2

I Scraped 40,000 Government Grants into Clean JSON. Here's What Broke.

DEV Community: python·Gala·1 day ago
#IyzuesDM
#devtopython#devto#dev#government#every
Reading 0:00
15s threshold

There are billions of euros in European NextGen funds and regional government grants sitting in Spain's National Subsidies Database (BDNS).

Finding them through the official portal, however, feels like navigating an intranet built in 2004 that actively wants you to give up.

Session timeouts every twelve minutes. Mystery search filters that return zero results if you misplace an accent mark. And tabular views where a single grant spans three nested IFRAMEs.

I wanted to make this data actually searchable for humans. So I set out to scrape the entire portal, normalize 40,000+ public grants into clean JSON, and serve them on a lightweight web interface: Subvenciones BDNS .

Naturally, almost every assumption I had about public web data immediately blew up in production.

Here is what broke, why modern APIs still struggle with legacy government systems, and how I built a declarative parsing engine in Python and PHP that runs reliably without constant maintenance.

TL;DR:

Public sector databases are not…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.