Tekkscope
Agentic search infrastructure that fails loudly instead of quietly.
trace · task 88f2
4.2s · 6 callssuccess
0.86
tool prec.
0.91
cost/query
$0.03
Per-stage tracing
Plan, retrieval and synthesis are timed and scored separately rather than as one opaque run.
Failing stage
Grounding fell below threshold. The trace names the stage, so debugging is not guesswork.
Cost per query
Tracked as a first-class constraint rather than discovered on the monthly bill.
Hover a marker to see what each part of the interface does.
Task success rate
01{{TASK_SUCCESS}}
Tool call precision
02{{TOOL_PRECISION}}
Median steps per task
03{{MEDIAN_STEPS}}
Cost per resolved query
04{{COST_PER_QUERY}}
The problem
What was broken before any of this existed.
Agentic search demos look excellent and production agents drift. They retrieve confidently wrong sources, loop, or silently degrade after a model or index change.
Engineering teams putting agent-driven search in front of real users.
Constraints I worked inside
ownership
end to end
role span
of four tracks
shipped
reached users
What changed
The same job, before and after the work.
Before
- Agents drifted silently after model changes
- One end-to-end score, no way to debug it
- Unbounded loops discovered on the bill
- Failures attributed vaguely to "the agent"
After
- Retrieval, tool choice and synthesis scored separately
- Fixed replay scenarios run on every change
- Hard caps on planning depth and tool calls
- Regressions surface in CI with the failing trace
The system, end to end
How a request travels through it.
query decomposed into a bounded set of sub-goals
tool calls executed with per-stage tracing
results checked against source before synthesis
reliability harness replays fixed scenarios on every change
stage-level regressions surface in CI with the failing trace attached
Where the effort went
Effort split and how quality moved.
$stat --effort
Where the time actually went
- Reliability harness45%
- Agent orchestration33%
- Tracing and alerts22%
$plot --trend
How it moved
Decisions
Every call I made, and the tradeoff I took for it.
Why
Unbounded agents fail in the most expensive way possible: slowly, and with a large bill.
Tradeoff accepted
Genuinely hard queries get cut off rather than solved.
Weighting
gaincostmy judgement, not a measurement
Why
A single end-to-end score tells you something broke but not what, which makes debugging guesswork.
Tradeoff accepted
More instrumentation to build and maintain before any of it pays off.
Weighting
gaincostmy judgement, not a measurement
Why
Comparability matters more than realism when the question is whether this change made things worse.
Tradeoff accepted
The suite drifts from real usage and needs deliberate refreshing.
Weighting
gaincostmy judgement, not a measurement
Outcome
What was true after it shipped.
Agent regressions became visible at the stage level in CI rather than after users noticed.
Cost per resolved query became a design constraint rather than a monthly surprise.
The reliability harness generalises to other agent products, which is where the work is heading next.
What worked
Treating agent reliability as an infrastructure problem instead of a prompt problem.
What was hard
Scenario design. Writing tasks that are hard enough to be informative but stable enough to compare took several rewrites.
What I would change
I would build the trace viewer earlier. Reading agent failures in raw logs slowed every debugging session.
$./next-step
Hiring for a product engineer who can carry the whole thing?
Send me the role and the problem behind it. I will tell you honestly whether I am the right fit.