My Crawler Now Tells You What It Missed

The worst failure mode of a data pipeline is not an error. It's silent junk: a 404 page filed as content, a recipe with the ingredients dropped, a forum thread quietly missing its replies — all returned as "success." If you feed LLMs for a living, you have eaten this. I have shipped this.

PageSift 0.80 is the release where I tried to eliminate that failure mode from my own engine. The theme is not speed this time (0.70 was speed). The theme is: results should say what they are, including what they failed to get.

1. Results that confess

Extraction is heuristic and always will be. The bug is pretending otherwise.…