Menu

Your scraper will not throw an error: designing page extraction that fails loudly, one field at a time
📰

Your scraper will not throw an error: designing page extraction that fails loudly, one field at a time

DEV Community: python·MapToSite·1 day ago
#5sydfr2f
#devtopython#devto#dev#field#tier
Reading 0:00
15s threshold

A scraper almost never fails the way a program is supposed to fail. It does not raise. It returns a number.

Last week a price list came back as 0 . The page had been redesigned: the price was still on it, in a <span> that no longer carried the class the extractor looked for. The extractor found nothing, applied .get_text() to nothing, and the pipeline wrote 0 into a table, where 0 is a legal price. Nobody saw the problem until a quote went out at zero.

That is the failure mode worth engineering against, and it is not the one most scraping guides prepare you for. An exception is a gift: it stops, it names a line, it leaves a hole you notice.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.