$ls ./writing

Notes on systems that have to work

The parts of AI engineering that do not demo well: evaluation, reliability, and deciding what not to build.

topicsRetrievalEvaluationAgentsReliabilityProductCareer3 pieces

Build the eval before you touch the retriever

Most retrieval work goes wrong in the same place: teams optimise before they can measure, then argue about taste for a quarter.

An evaluation that depends on someone remembering to run it stops being run by week three.

from “Build the eval before you touch the retriever”

More notes

Traditional testing assumes determinism. Agent systems break that assumption, so reliability has to be designed rather than asserted.

The title is fashionable and vague. Here is the concrete version: you own the problem, the tradeoff, and the number that says whether it worked.

$./next-step

Most of what I have learned is easier to explain than to write down.