⚠️Need of Hybrid Search

We have documents containing error codes in python with their respective definitions and use-cases. User writes a query to know about "What is ERR_404_AUTH?"

Classic Rag: Will retrieve all the authentication and error related context it can find from vector db (document embeddings).

Lexical search: Will search for terms ["What", "is", "ERR_404_AUTH"]

Hybrid search: Will search for keyword "ERR_404_AUTH" and retrieve semantically similar documents using similarity search.

🛠️Using BM25

Take BM25 as extended version of TF-IDF for key-word based search.

The step-wise implementation of BM25 in LangChain is straightforward because LangChain provides a built-in BM25Retriever .

Here is the step-wise implementation alongside the intuition.

# pip install rank_bm25 from langchain_community.retrievers import BM25Retriever from langchain_core.documents import Document

# Chunks from your text splitter chunks = [ Document ( page_content = " The AX-705 engine uses a 4-stroke cycle.…