Series: VectorRAG and GraphRAG, Hands-On Part 1: Retrieval-Augmented Generation - Offline Indexing pipeline · Part 2: Online retrieval pipeline and measure with evals (you are here) · Part 3: GraphRAG, When Connections Matter (Coming next)
Source code: rag-project. Every retrieval number in this post comes from running that code.
Pipeline 2, querying: retrieve, then generate
Now the query pipeline. The index is built, and ask.py uses it for every question:
Embed the question with the same model, adding the BGE question prefix.
Search the index.
--modepicks naive vector search, keyword, hybrid or hybrid + reranker. following sections explain the last three.Build the prompt from the top chunks, each with its id, title, heading and date.
Generate: send it to Claude. With
--no-llm(or no API key) it stops after step 3 and just shows what it found.
Steps 1 to 3 run on your laptop; only step 4 calls an API.
The prompt is plain: instructions, the retrieved chunks with their ids and dates, then the question.
Refer code: # rag/llm.py
Each line of that system prompt does a job:
“Use ONLY the context” closes the book on memory.
“Cite chunk ids” makes the answer checkable.
“Prefer the most recently updated” handles conflicting versions, since every chunk shows its date.
“Say I don’t know” gives the model a respectable exit instead of guessing. That one line prevents many hallucinations.
complete() is a thin wrapper over the official anthropic SDK. It uses Claude Opus 5.5 at low effort, which is plenty for answering from given text.
Let’s ask our first question:
$ python ask.py "How many earned leave days can I carry forward?"
Mode: naive Question: How many earned leave days can I carry forward?
1. HR-POL-007#2 score=0.832 Leave Policy > Earned leave (EL)
2. HR-POL-007#4 score=0.690 Leave Policy > Other leave
3. HR-POL-007#3 score=0.684 Leave Policy > Sick leave (SL)
4. HR-POL-014#2 score=0.608 Travel Policy > Mode of travel
Answer:
You can **carry forward up to 15 days** of unused earned leave into the next year. Any unused EL above 15 days lapses on 31 December [HR-POL-007#2].
Your total EL balance can never exceed **45 days** at any time [HR-POL-007#2].This is plain vector search (naive, the default mode), followed by one Claude call. The score is the cosine similarity between the question and each chunk. The right chunk is first, with a clear margin. When retrieval works this cleanly, the answer is easy for any good LLM.
But one good question proves nothing. How do we know retrieval works for the next hundred questions?
Before improving anything: measure, the evals
An LLM is probabilistic, so you need checkers: checks that tell you, automatically, whether things work. For RAG, the first checker is a small evaluation set.
Ours is 13 questions in data/eval_set.json, deliberately of different kinds:

For each question I listed the key facts any correct answer must use, such as "15 days", "Mary Fernandes" or "₹1,00,000". The metric is key-fact recall: what share of those facts made it into the chunks we hand to the LLM?
Refer code: # rag/evaluation.py
Why measure retrieval and not the final answer? Because if the facts aren’t in the context, no model can answer correctly. Retrieval recall is cheap, deterministic, needs no API calls, and can run in CI on every change. Its blind spot: a fact can appear by coincidence in an unrelated chunk.
For answer quality, the standard metrics are the four that RAGAS made popular:
faithfulness: is every claim in the answer supported by the context?
answer relevance
context precision
context recall
They need an LLM judge, so they cost money on every run. Start with retrieval recall, and add an answer judge once retrieval is good.
One warning the metrics won't give you: a system can score 0.95 on faithfulness and still give wrong business answers if the documents themselves are stale. Faithful to an outdated policy is still wrong. Metrics check the pipeline; someone still has to own the documents.
Here is our baseline, naive vector search on raw chunks, 4 chunks per question:
0.82 sounds decent until you read it as "roughly one fact in six that the answer needs never reaches the LLM". Each fix changes one of the two pipelines. That decides how you try it.

An indexing fix means rebuilding the index; the question side stays the same. A querying fix uses the same index and searches it differently.
Fix 1: give every chunk its address
Look at Q7: "What was the root cause of INC-2026-0815?" The incident report has a section called "Root cause". Here it is as a chunk:
A configuration change on 12 August set log retention to 30 days instead of 7 days on three high-volume topics. Disk usage alerts were configured, but they were routed to an email list that nobody monitored.
Do you see the problem? The chunk never says “INC-2026-0815”, and never says “root cause”. Those words are in the document title and the heading, and chunking cut them off. It’s the train coach without its train number.
Here is what plain vector search finds for that question, on chunks exactly as split (the bluepeak_raw index):
$ python ask.py --no-llm --index bluepeak_raw “What was the root cause of INC-2026-0815?”
1. PRJ-FALCON#3 score=0.563 Project Falcon - Project Charter > Technology
2. PRJ-FALCON#5 score=0.557 Project Falcon - Project Charter > Key risks
3. INC-2026-0815#4 score=0.553 Post-incident Review - INC-2026-0815 Orbit Outage > Actions
4. IT-POL-003#2 score=0.548 IT Security Policy > Reporting incidentsThe “Root cause” chunk isn’t in the top four. Two Falcon chunks beat it because they mention Orbit, the platform that went down. The root-cause chunk mentions neither Orbit nor the incident.
The fix: prefix every chunk with its address (Document title > Section heading) before embedding and keyword indexing. This is a change to pipeline 1: 02_index.py builds a second index, bluepeak_titled, where the text that gets embedded for the root-cause chunk starts with Post-incident Review - INC-2026-0815 Orbit Outage > Root cause. The LLM still sees the original chunk text. The query side doesn’t change at all; --index only picks which index to search.
def titled(chunks):
"""Give every chunk its 'address': document title and section heading."""
return [f"{c.title} > {c.heading}\n{c.text}" for c in chunks]
build_index(chunks, name="bluepeak_titled", texts_to_embed=titled(chunks))Same question, same plain vector search, titled index, check the above code, its the default index selected.
$ python ask.py --no-llm "What was the root cause of INC-2026-0815?"
1. INC-2026-0815#4 score=0.657 Post-incident Review - INC-2026-0815 Orbit Outage > Actions
2. INC-2026-0815#3 score=0.613 Post-incident Review - INC-2026-0815 Orbit Outage > Root cause
3. INC-2026-0815#2 score=0.598 Post-incident Review - INC-2026-0815 Orbit Outage > Impact
4. INC-2026-0815#5 score=0.588 Post-incident Review - INC-2026-0815 Orbit Outage > LessonsThe root-cause chunk is now second, and all four chunks come from the right document.
The Spark team chunk (”Tech lead: Kavya Reddy…” with no “Spark” in it) shows the same effect. Asked “Who is the tech lead of Project Spark?”:
Raw index: the Team chunk comes second, at 0.612, almost tied with three other chunks (0.621, 0.610, 0.607). It makes the top four here, but a slightly different wording of the question could push it out.
Titled index: it comes first, at 0.752, well clear of the next “Team” chunk, Ledger’s (0.631).
Across the eval set, titles fixed exactly the questions they were meant to fix: Q7 went from 0.50 to 1.00 and the global question Q13 from 0.20 to 0.60.
Fix 2: hybrid search
Embedding search is great at meaning and weak at exact strings. To an embedding model, “SEC-114”, “HR-POL-012” and “INC-2026-0815” are nearly meaningless tokens. Real users type exactly these things: policy numbers, error codes, product SKUs, ticket IDs.
The old-school keyword search, BM25, is the opposite. It scores chunks by how often the question’s exact words appear, weighted by how rare those words are. It finds “INC-2026-0815” instantly but doesn’t know “broadband” means “internet”.
So run both and combine. That’s hybrid search, and current guidance treats it as the default, not an advanced trick.
Combining two rankings: Reciprocal Rank Fusion
You can’t add a cosine similarity (0 to 1) to a BM25 score (0 to 11): different scales. Reciprocal Rank Fusion (RRF) ignores the scores and uses only the ranks.
A chunk that ranks well in both lists rises to the top. A chunk that ranks first in one list and nowhere in the other still gets a good score. The constant 60 comes from the original RRF paper and is the common default.
Fix 3: reranking
Embedding search is fast because the question and each chunk are embedded separately: you pre-compute every chunk’s vector once. This is called a bi-encoder. The price is that the model never sees the question and the chunk together, so subtle relevance gets lost.
A cross-encoder (reranker) reads the question and one chunk together and outputs a relevance score. It’s much sharper and much slower, too slow to run over every chunk. So we use it in two stages, like campus placements: a quick resume screen for everyone, then a proper interview for the shortlist.
In short:
Stage 1, shortlist: hybrid search returns the 20 best candidates. It’s fast because the chunk vectors were computed during indexing.
Stage 2, rerank: the cross-encoder reads the question and each candidate together, scores all 20, and we keep the best 4. Nothing in the index changes; this is purely a querying fix.
Here it is on Q11, a question that needs facts from more than one document:
$ python ask.py --no-llm --mode hybrid "Which delivery heads own the client accounts that were affected by the Orbit outage?"
1. INC-2026-0815#2 score=0.032 Post-incident Review - INC-2026-0815 Orbit Outage > Impact
2. INC-2026-0815#5 score=0.031 Post-incident Review - INC-2026-0815 Orbit Outage > Lessons
3. INC-2026-0815#3 score=0.031 Post-incident Review - INC-2026-0815 Orbit Outage > Root cause
4. INC-2026-0815#1 score=0.030 Post-incident Review - INC-2026-0815 Orbit Outage > Summary
$ python ask.py --no-llm --mode rerank "Which delivery heads own the client accounts that were affected by the Orbit outage?"
1. ORG-001#3 score=0.074 Bluepeak Tech Solutions - Organisation Structure > Business units
2. ACC-MRB#1 score=0.032 Account Profile - Meridian Bank > Account Profile: Meridian Bank
3. ORG-001#4 score=0.030 Bluepeak Tech Solutions - Organisation Structure > Internal units
4. ACC-MRB#3 score=0.030 Account Profile - Meridian Bank > Relationship notesHybrid search filled all four slots with the incident report, because every one of its chunks shares words and meaning with “Orbit outage”. But the question asks who the delivery heads are, and the incident report doesn’t say. The reranker read each candidate together with the question and pulled in the chunks that name delivery heads: the organisation structure and the Meridian Bank account profile. (The scores aren’t comparable between the two runs: hybrid shows fusion scores, rerank shows the cross-encoder’s relevance scores.)
Key-fact recall went from 0.50 to 0.75. Hybrid search found the affected projects (Falcon, Spark) but no delivery heads; the reranker found both delivery heads, Mary Fernandes and Farhan Qureshi. It isn’t perfect: the Impact chunk, which names Spark as affected, dropped out. Four chunks can’t hold every link of this chain, and that is the kind of question Part 3 is about.
Across published benchmarks, reranking is usually the single biggest improvement you can add. The common pattern is to rerank the top 50–100 candidates and keep 5–10.
Fix 4: contextual retrieval
Adding titles gave each chunk a basic address. Contextual Retrieval, published by Anthropic, goes further. For every chunk, ask an LLM to write one or two sentences that place the chunk within its whole document. Prepend that context before embedding and before BM25 indexing.
The trick that makes this affordable is prompt caching. The whole document goes first in the prompt and is marked for caching. The second, third and fourth chunks from the same document reuse it at a fraction of the price. You pay for each document roughly once, plus a small amount per chunk.
We will continue with next part on why GraphRAG is useful and what problem it solves, Coming next.





