Menu

1 / 2

One Engine, Seven Open-Source Projects: PageSift 1.4 and the yolow Family

DEV Community: opensource·ゴエンチョウ·2 days ago
#1pvqIUjO
#devtoopensource#devto#dev#pagesift#crates
Reading 0:00
15s threshold

PageSift started as the ingestion layer of an AI research platform I was building: something had to fetch web pages and documents and turn them into clean Markdown that an LLM pipeline could actually use. It grew into a self-hosted web scraper, crawler and document parser — Firecrawl- and Jina-compatible API, MCP server, Python/Node SDKs, one Rust binary.

Over the 1.x series I kept doing one thing: every time a piece of the engine turned out to be generally useful, I cut it out and gave it its own repository, its own versioning, and an MIT license. PageSift 1.4 finishes that split — and adds the subsystem the extraction was making room for.

PageSift 1.4

The split, completed. Wave one (1.2.7) turned the base64, PDF, HTML and content crates into thin shims over standalone libraries. Wave two (1.4.0) does the same for the remaining three: pagesift-office → yolowdoc , pagesift-syntax → yolowsyntax , pagesift-compress → yolowzstd .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.