No CSS selectors, no headless browser. The data was in the HTML all along.
This week I built one that collects upcoming events in a city from Eventbrite, Meetup and Luma and puts them in one list. I expected three scrapers full of fragile selectors. I ended up writing almost none, because all three sites already ship their event data inside the page, in a form a program can read. Each one just keeps it somewhere different.
π¦ First, the rules
Before writing any code I read each site's robots.txt, keeping only the rules that apply to every bot ( User-agent: * ):
Eventbrite blocks its RSS and Atom feeds, checkout links and a few API paths. The city search pages ( /d/... ) and event pages are open.
Meetup blocks its API paths ( /api , /meetup_api ) and its feeds. The search page ( /find/ ) and event pages are open.
Luma 's robots.txt only has rules for Googlebot.
robots.txt isn't the whole story, so read each site's terms as well before you build on it.β¦