honestly I just spent the last 3 weeks trying to compile a clean historical pricing dataset for a machine learning project, and I swear every major real estate site is locked down like fort knox now.

I used to just run basic python scrapers overnight, but now you hit aggressive rate limits or those endless cloudflare loops after like 50 requests. It's exhausting. I literally had to route my whole pipeline through residential proxies just to keep the sessions alive long enough to pull the raw tables without my ip getting permabanned. Kinda insane how many hoops we have to jump through just to aggregate public info nowadays.

Anyway I finally managed to scrape about 800k rows across the east coast. Still cleaning up some messy geocoding stuff but I should be able to drop a mega link or github repo here by the weekend if anyone wants to play around with it. anyone else noticing how brutal scraping has gotten this year?