Building autonomous deep research pipelines (like Stanford STORM or OpenAI Deep Research) typically faces two massive hurdles:
Insane Token Burn: Ingesting dozens of raw web pages consumes 200k to 500k tokens per research question, costing $5 to $15 per report.
Hallucinated Citations: When LLMs summarize long contexts, they frequently misattribute quotes or fabricate URLs and reference numbers.
To solve this, we engineered agent-deep-research — a local-first Python engine that slashes token consumption by over 85% while guaranteeing zero hallucinated citations through cryptographic SHA-256 provenance ledgers.
The Core Innovation: BM25 Lexical Distillation
Instead of feeding raw HTML or giant text blobs into the LLM context:
The engine fetches candidate sources using free, zero-key endpoints (DuckDuckGo, Wikipedia, ArXiv).
Content is partitioned into 250-word passages and indexed in an in-memory BM25 Lexical Matrix .
Only the top-k highest density factual passages matching the query sub-claims are…