I Process All of English Wikipedia in Under 4 Minutes
After a one-time 25 GB download, every subsequent extraction — jazz musicians, sci-fi novels, literary authors, whatever you want — runs against all 25.8 million Wikipedia pages in about three and a half minutes. Here's how I got there.
The goal
Transform a corpus of semi-structured Wikipedia data into searchable, structured database tables as fast as possible .
Background
Previously, I've done data visualization projects with data scraped from Wikipedia pages. I used asynchronous, depth-first page crawling that extracted key data related to jazz music from artist, album, and genre pages.
After fetching a page, a top-level switch classified each one as a jazz artist, jazz album, or jazz-related genre. This determined which shape of Beautiful Soup -based extractor to run on that page, each one suited to target a specific table in the extracted database (artist, album, or genre).…