Retrievers & Re-Ranking

Basic vector search gets you started, but users ask vague, short or keyword-heavy questions. Advanced retrievers fix this. In this session, let us see six techniques that make retrieval much more reliable, and calculate hybrid search scores by hand.

01Multi-query retriever

A short question like “FD early exit?” may miss chunks that use different words. A multi-query retriever asks the LLM to rewrite the question in several ways, searches with each, and merges the results.

from langchain.retrievers.multi_query import MultiQueryRetriever

mq = MultiQueryRetriever.from_llm(retriever=db.as_retriever(), llm=llm)   # db and llm from Session 31
docs = mq.invoke("FD early exit?")

02Multi-vector, parent-document and self-query

• Multi-vector retriever: store several vectors per document, such as a summary, likely questions, and the raw chunks. Search any of them, return the original.
• Parent-document retriever: the parent–child idea from Session 33: search small chunks, return the bigger parent.
• Self-query retriever: the LLM turns the question into a search query plus a metadata filter.

03Contextual compression

Retrieved chunks often contain only one useful sentence. Contextual compression trims each chunk to the parts relevant to the question (using an LLM or an embedding filter) before sending it to the generator. Fewer tokens, less noise, lower cost.

04Hybrid search: BM25 + dense

Dense (vector) search understands meaning but can miss exact terms like product codes or section numbers. BM25 keyword search is great at exact terms but misses synonyms. Hybrid search runs both and merges the rankings, often with Reciprocal Rank Fusion (RRF):

def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, doc in enumerate(ranking, start=1):
            scores[doc] = scores.get(doc, 0) + 1 / (k + rank)
    return sorted(scores.items(), key=lambda x: -x[1])

bm25  = ["FD_rules_p4", "Savings_p2", "FD_faq_p1"]          # keyword search ranking
dense = ["FD_faq_p1", "FD_rules_p4", "Loan_p9"]             # vector search ranking
for doc, s in rrf([bm25, dense]):
    print(f"{doc:12s} {s:.4f}")

FD_rules_p4 was 1st in BM25 and 2nd in dense, so it wins overall. Documents found by both methods rise to the top. In LangChain:

from langchain_community.retrievers import BM25Retriever
from langchain.retrievers import EnsembleRetriever

bm25 = BM25Retriever.from_documents(chunks); bm25.k = 5
dense = db.as_retriever(search_kwargs={"k": 5})
hybrid = EnsembleRetriever(retrievers=[bm25, dense], weights=[0.5, 0.5])   # weighted RRF
docs = hybrid.invoke("Section 3.2 premature withdrawal")

05Re-ranking

Retrievers are fast but rough: they embed the question and the chunk separately. A cross-encoder re-ranker reads the question and each chunk together and gives a much more accurate relevance score. The usual pattern: retrieve 20–50 candidates quickly, re-rank, keep the top 3–5.

# pip install sentence-transformers
from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-base")
question = "Penalty for breaking an FD early?"
candidates = [d.page_content for d in hybrid.invoke(question)]
scores = reranker.predict([(question, c) for c in candidates])
top4 = [c for _, c in sorted(zip(scores, candidates), reverse=True)[:4]]

06Summary

• Multi-query rewrites vague questions; self-query adds metadata filters.
• Multi-vector and parent-document retrievers search small, return big.
• Compression trims chunks to what matters.
• Hybrid search (BM25 + dense, merged with RRF) catches both meanings and exact terms.
• Re-ranking with a cross-encoder gives the most accurate final top-k.

07Quick quiz

08References

• Reciprocal Rank Fusion (Cormack et al., 2009)
• LangChain retrievers
• BGE re-rankers