Menu

Scrape Wikipedia in 4m♭ (four minutes flat)
📰

Scrape Wikipedia in 4m♭ (four minutes flat)

DEV Community: datascience·wright-benjamin-1701·16 days ago
#GmzMh3Gx
#devtodatascience#science#devto#dev#jazz
Reading 0:00
15s threshold

I Process All of English Wikipedia in Under 4 Minutes

After a one-time 25 GB download, every subsequent extraction — jazz musicians, sci-fi novels, literary authors, whatever you want — runs against all 25.8 million Wikipedia pages in about three and a half minutes. Here's how I got there.

The goal

Transform a corpus of semi-structured Wikipedia data into searchable, structured database tables as fast as possible .

Background

Previously, I've done data visualization projects with data scraped from Wikipedia pages. I used asynchronous, depth-first page crawling that extracted key data related to jazz music from artist, album, and genre pages.

After fetching a page, a top-level switch classified each one as a jazz artist, jazz album, or jazz-related genre. This determined which shape of Beautiful Soup -based extractor to run on that page, each one suited to target a specific table in the extracted database (artist, album, or genre).…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.