{{POST_BODY}} Replace this with the full post. The outline below is the argument I would make.
Every retrieval project starts with a change that feels obviously good. A bigger embedding model, a smarter prompt, a chunking tweak. The change ships, the answers feel different, and nobody can say whether the product improved.
The fix is unglamorous. Assemble a labelled query set from real usage before making a single retrieval change, define what relevant means for your product in writing, and put the numbers in CI so nobody has to remember to run them.
The cost is real: two weeks with nothing visible shipped. The return is that every argument after that ends in a number instead of an opinion.