I wanted a real coverage number for our own appliance database, not a vibe. So I crawled it.

The trigger was boring. Someone asked for a Maytag dryer fault code and landed on a page that listed twelve washing machines and mentioned, in passing, that one of them had a code list. Their model wasn't in the twelve and wasn't a washer. The page wasn't lying. It just wasn't useful, and the gap between those two things is what I wanted to measure.

The crawl

Our sitemap index for models held 4,800 URLs. 817 are brand × appliance index pages, and each one embeds a tally of itself in prose:

"9 Kenmore clothes dryers documented. Together they carry 0 published replacement part numbers, and 0 of them have a confirmed error-code list."

Enter fullscreen mode

Exit fullscreen mode

That's three integers per page, so a full catalogue total costs 817 requests instead of 8,000. Roughly:

PAT = re . compile ( r " (\d+) .{0,40}? documented\.…