Berribot Retrieval Evaluation
Turning "the answers feel better" into a number the team could trust.
retrieval eval · run 214
passingnDCG@10
0.81
+.09
MRR
0.74
+.06
Recall@20
0.92
+.04
Tracked metrics
nDCG@10, MRR and Recall@20 with the delta against the last run, so a change is visible immediately.
Per-commit history
Every bar is one CI run. Regressions show up as a dip before anyone files a support ticket.
Config comparison
Hybrid and reranked configurations scored against the dense-only baseline on the same query set.
Hover a marker to see what each part of the interface does.
nDCG@10 lift
01{{NDCG_LIFT}}
hybrid + rerank vs dense-only baseline
Recall@20
02{{RECALL_AT_20}}
on the labelled eval set
p95 latency
03{{P95_LATENCY}}
after reranking was added
Eval set size
04{{EVAL_SET_SIZE}}
query / passage pairs
The problem
What was broken before any of this existed.
The assistant shipped fast and answered well enough in demos, but nobody could say whether a change made it better. Every prompt edit turned into an argument, and regressions only surfaced when a customer complained.
Internal engineers shipping changes weekly, and the end users asking questions against a growing document corpus.
Constraints I worked inside
ownership
end to end
role span
of four tracks
shipped
reached users
What changed
The same job, before and after the work.
Before
- Quality was argued, not measured
- Prompt changes shipped on instinct
- Regressions found by customers
- No baseline to compare against
After
- Retrieval quality tracked per commit
- Hybrid rewrite approved on evidence
- Regressions caught in CI
- A labelled golden set the team maintains
The system, end to end
How a request travels through it.
normalised, expanded, routed by intent
BM25 and dense candidates merged with reciprocal rank fusion
cross-encoder scores the top candidates
grounded generation with citation spans
nDCG@k, MRR, Recall@k logged to Weights & Biases per run
Where the effort went
Effort split and how quality moved.
$stat --effort
Where the time actually went
- Building the eval set40%
- Retrieval rewrite34%
- CI and observability26%
$plot --trend
How it moved
Decisions
Every call I made, and the tradeoff I took for it.
Why
Without a baseline, every improvement is a story. With one, the argument ends in a number.
Tradeoff accepted
Two weeks with nothing visible to show. I traded early momentum for the ability to measure everything that followed.
Weighting
gaincostmy judgement, not a measurement
Why
Failure analysis showed most misses were exact-term queries: product codes, error strings, proper nouns. Lexical search fixes those cheaply.
Tradeoff accepted
More moving parts in the retrieval path, and two indexes to keep in sync instead of one.
Weighting
gaincostmy judgement, not a measurement
Why
Reranking is where the quality lift lives, but it is also the most expensive step per query.
Tradeoff accepted
A relevant document sitting deep in the candidate list stays buried. I accepted that in exchange for staying inside the latency budget.
Weighting
gaincostmy judgement, not a measurement
Why
An evaluation that depends on someone remembering to run it stops being run by week three.
Tradeoff accepted
Slower CI, and the eval set itself became something the team had to maintain.
Weighting
gaincostmy judgement, not a measurement
Outcome
What was true after it shipped.
Retrieval quality became a tracked metric with a per-commit history instead of a vibe.
The hybrid and reranking rewrite shipped with evidence attached, which is why it got approved quickly.
Structured logging and latency alerting meant regressions were caught in CI rather than in support tickets.
What worked
Leading with measurement. Once the numbers existed, prioritisation arguments got much shorter and the team stopped shipping on instinct.
What was hard
Labelling quality. Early relevance judgments were inconsistent until I wrote down what "relevant" actually meant for this product.
What I would change
I would add an LLM-as-judge pass for answer faithfulness earlier, so retrieval quality and answer quality could be separated from week one.
$./next-step
Hiring for a product engineer who can carry the whole thing?
Send me the role and the problem behind it. I will tell you honestly whether I am the right fit.