Ask most teams how they evaluate their agent and the answer is some version of: "we have a set of test prompts, we check if the final answer is correct, and we track the pass rate."

This is a reasonable way to evaluate a single LLM call. It is not a reasonable way to evaluate an agent.

An agent is not a function that maps input to output. It's a sequence of decisions — which tool to call, what parameters to pass, when to ask a human, when to keep going — where each decision shapes what happens next, and where the same starting point can legitimately produce different but equally valid paths to the goal. Grading only the final answer is like grading a road trip by whether the car eventually arrived, with no interest in whether it took the highway, got lost twice, or ran three red lights along the way.

If your eval only looks at the destination, you will ship agents that get lucky on your test set and fail in production in ways your eval never had a chance of catching. This issue is about building an evaluation methodology that actually reflects how agents work — trajectories, not just outputs — and rigorous enough that you can trust it when the moment comes to change your model.

Why Output-Only Evaluation Fails

Consider an agent tasked with rescheduling a customer's flight. The final output looks correct: "Your flight has been rebooked for March 15th, confirmation number ABC123." Output-only evaluation marks this a pass.

What output-only evaluation doesn't see: the agent called the cancellation API before confirming the new flight had availability. It got lucky — the new flight had a seat. Run the same scenario with a fuller flight, and the same agent cancels a customer's ticket and then fails to rebook them. The trajectory was wrong. The output, in this one instance, happened to look right anyway.

This is the central problem with grading only the final answer: it cannot distinguish between an agent that reasoned correctly and an agent that got lucky. And agents that get lucky in eval get unlucky in production, at scale, in front of real users, usually at the worst possible moment.

Trajectory Evaluation: Grading the Path, Not Just the Destination

Trajectory evaluation looks at every decision point in the agent's execution — not just the final output — and scores it independently.

Tool selection accuracy asks: at each point where the agent chose to call a tool, was that the right tool for the situation? This needs to be measured per-step, not aggregated into a single number. An agent that's 95% accurate on tool selection overall but consistently picks the wrong tool for one specific scenario type has a real, fixable problem that an aggregate score conceals entirely. Break this down by tool and by task category. The aggregate number is where regressions go to hide.

Parameter accuracy is the layer most teams skip, and it's where a huge share of real production failures live. The agent chose the correct tool — good. Did it pass the correct parameters? A date range that's off by one day. A customer ID pulled from the wrong turn of the conversation. A search query that's technically related but misses the actual intent. These failures are invisible if you only check whether the tool call happened and whether the final output reads plausibly — the model is often good enough to paper over a parameter error with confident, fluent language. You have to check the parameters themselves, structurally, against what the task actually required.

Step efficiency captures something output evaluation completely misses: how did the agent get there? Two agents that both reach the correct final answer are not equivalent if one took four steps and the other took fourteen, looping through the same tool with slightly different parameters until something worked. Step efficiency is simultaneously a quality signal (looping often indicates the agent doesn't understand why its calls are failing) and a cost and latency signal hiding inside your trajectory data. Track it.

Recovery behavior is what happens when something goes wrong mid-trajectory — a tool call fails, returns unexpected data, or times out. A well-built agent recognizes the failure and responds sensibly: retries with adjusted parameters, falls back to an alternative tool, or asks the user for clarification. A poorly built agent — or one under evaluation pressure — sometimes hallucinates a plausible continuation as if the failed call had succeeded. This is one of the most dangerous failure modes in production agents, and it is entirely invisible unless your eval deliberately injects tool failures and checks how the agent handles them.

Human-in-the-Loop Accuracy: Escalating Exactly Enough

If your agent has HITL gates for irreversible or high-impact actions — and per a prior issue, it should — the calibration of those gates is itself something that needs rigorous evaluation, with its own precision and recall.

Escalation precision measures: of every time the agent escalated to a human, how many escalations were actually warranted? Low precision means the agent cries wolf constantly — routine actions get flagged for approval, humans get trained to rubber-stamp requests without reading them carefully, and the entire safety mechanism degrades into theater. Over-escalation is not the safe failure mode it appears to be. It quietly destroys the value of the HITL gate by training humans to stop paying attention.

Escalation recall measures the more dangerous direction: of every situation that genuinely warranted a human checkpoint, how many did the agent actually escalate? This is the metric that catches the agent quietly taking an irreversible action it should have paused on. Recall failures here are the ones that produce real incidents — and because they represent things that didn't happen (a needed escalation that didn't occur), they're structurally harder to notice than precision failures. You have to build eval scenarios specifically designed to test whether the agent recognizes situations that should trigger escalation, not just check the escalations that did happen.

Approval request quality is a softer but real metric. When the agent does escalate, is the request useful? "Should I proceed?" burdens the human with reconstructing context. "Delete these 47 records identified by these criteria — confirm?" gives the human exactly what they need to make a fast, informed decision. Poor escalation quality doesn't show up as a wrong answer, but it shows up as slower human response times and worse approval decisions, both of which degrade the overall system.

Building the Right Dataset: This Is Where Most Eval Efforts Fail

Everything above assumes you have a dataset capable of surfacing these failure modes. Most teams don't, because most teams build eval datasets the way they'd build a unit test suite — a fixed script of inputs and expected outputs. That approach breaks down completely for multi-turn agent evaluation, for the reason you identified: a fixed script assumes the conversation follows a predetermined path, and agent conversations don't.

Each eval entry needs to be a conversation tree, not a script. If turn one of the eval scenario is "I need to reschedule my flight," the agent might respond by asking which flight, or by asking for the new preferred date, or by pulling up the customer's itinerary directly if it has access. A fixed script that only has one predetermined "turn two" for the user to send cannot follow all three of these paths. The eval framework needs to dynamically select the next user message based on the agent's actual trajectory so far — which requires either a human evaluator following branches live, or a simulated user model sophisticated enough to respond appropriately to whatever path the agent actually took, not just the path the dataset author anticipated.

This is a meaningfully harder engineering problem than static eval sets, and it's exactly why most teams don't do it — and exactly why their eval scores don't predict production behavior. Building a simulated user that can respond coherently to the actual state of a branching conversation, rather than reciting a pre-written line, is itself a small agent-building problem. Investment here pays for itself the first time it catches a regression a static script would have missed entirely.

The dataset must be representative of production usage, not representative of what's interesting to test. This sounds obvious and is routinely violated. Eval sets built by engineers tend to overrepresent edge cases, adversarial scenarios, and "interesting" failure modes, because those are the scenarios engineers find compelling to write. Real production traffic is overwhelmingly mundane — the same handful of task types, the same common conversation lengths, the same predictable distribution of user intents. If your eval set doesn't match that distribution, your eval score doesn't tell you how the agent will perform in production. It tells you how the agent performs on the specific slice of unusual scenarios your team found interesting to write down. Sample your eval set from real production logs, stratified to match actual usage distribution, supplemented with — but not dominated by — deliberately constructed edge cases.

Agents are stochastic. One rollout is noise, not signal. The same eval scenario, run against the same agent, can produce different trajectories on different runs — different tool call orderings, different phrasing, sometimes different outcomes entirely, especially at nonzero temperature. A single-rollout evaluation score is a sample of one from a distribution, and treating it as ground truth is a statistical error. Run each eval scenario multiple times — five is a reasonable floor for meaningful scenarios — and report the distribution of outcomes, not a single pass/fail. An agent that succeeds 4 out of 5 times on a scenario is meaningfully different from one that succeeds 5 out of 5 or 1 out of 5, and a single-rollout eval cannot distinguish between them.

Statistical Rigor: Trusting the Number Enough to Ship On It

This is the part of the outline that deserves the most emphasis, because it's the difference between an eval suite that's a useful internal signal and one that's a defensible basis for a production decision — including, critically, the decision to swap model versions or model families with confidence.

Sample size has to be large enough to detect the regression you actually care about. This is a statistical power question, not a vibes question. If you care about detecting a 2% regression in task success rate, you need to calculate — not guess — how many eval examples that requires at your desired confidence level. A 50-example eval set has enormous variance; a 2-3 percentage point swing in the observed pass rate can happen from sampling noise alone, with no actual change in underlying agent quality. Teams that ship model changes based on a 50-example eval run are often making decisions on noise. Do the power calculation. Size the eval set to the regression you need to catch.

Every reported metric should carry a confidence interval, not just a point estimate. "87% task success rate" is close to meaningless on its own. "87% ± 1.2%" and "87% ± 11%" represent very different levels of confidence in that number, and only one of them is a defensible basis for a ship decision. Build confidence interval reporting into your eval infrastructure as a default, not an occasional deep-dive.

Representativeness is what makes model swaps safe. This is the direct payoff of everything above. If your eval dataset is large, statistically powered, and genuinely representative of production traffic — not just a curated set of interesting cases — then a new model version or a different model family entirely can be evaluated against the same dataset, and the resulting score is actually predictive of how it will perform once you ship it. This is what lets you upgrade model versions, or even swap model families, with confidence rather than hope. Without that representativeness, every model swap is effectively a production experiment, discovered live, at your users' expense.

Slice your metrics or your aggregate will lie to you. A model swap that improves overall task success by 3% while silently breaking one specific tool category, one specific user segment, or one specific conversation length bucket is not a strict improvement — it's a trade a stakeholder needs to consciously make, not one that should happen invisibly inside an aggregate number. Always report metrics sliced by task type, by tool category, and by any dimension where you have reason to believe performance might vary. The aggregate is the last thing you look at, not the first.

Goal Success Rate: The Metric That Actually Matters

Everything above measures whether individual decisions within a trajectory were correct. Goal success rate measures something different and, ultimately, more important: across the entire multi-turn interaction, did the user's actual underlying need get met?

This distinction matters because an agent can make a series of individually defensible decisions and still fail the user's actual goal — asking one too many clarifying questions that frustrates the user into abandoning the task, technically completing the literal request while missing the underlying intent, or requiring so many corrections along the way that the interaction, while technically "successful," was a bad experience the user won't repeat.

Goal success rate should be evaluated at the level of the complete interaction, against the user's actual underlying objective — not against a rubric of "was each individual response acceptable." This usually requires either human evaluation or a carefully validated LLM-as-judge (more on this below) that has been given the full conversation and the actual goal, and asked a single question: did this get resolved, well, from the user's perspective?

Pair this with turn efficiency to goal — how many turns did it take to get there — and unprompted correction rate — how often did the user have to redirect or clarify. An agent with a high goal success rate but a high correction rate is succeeding despite itself, papering over comprehension failures with persistence. That's a real signal worth tracking separately, because it tells you where the agent is fragile even when the top-line number looks fine.

What You're Missing — A Few Additions

Your framework covers the core exceptionally well. Here's what I'd add to round it out into something closer to a complete production eval practice.

LLM-as-judge needs its own validation before you trust it. Most trajectory and goal-success scoring at scale relies on an LLM to do the grading, because human evaluation doesn't scale to the volume you need. But the judge model is itself an unvalidated component until you check it. Take a meaningful subset of judge-scored examples, have humans score the same examples independently, and measure inter-rater agreement between the judge and the humans. If agreement is weak, the judge's scores across your entire eval set are unreliable, no matter how confident-sounding the judge's reasoning looks. Recalibrate the judge prompt, or the rubric, until agreement is strong — and re-check this periodically, because judge models drift in behavior across their own version updates too.

Rubric-based, partial-credit scoring beats binary pass/fail. A response that's correct except for one wrong parameter is meaningfully different from a response that's completely wrong, and a binary pass/fail metric collapses that distinction, hiding real quality signal. Build graded rubrics for your key task types — weighted scoring across the dimensions that matter (correctness, completeness, tool efficiency, tone) — and track the distribution of scores, not just a pass rate.

Groundedness and hallucination need explicit measurement, separate from task success. An agent can technically accomplish the task while asserting a fact that isn't actually supported by any tool result it retrieved — confidently citing a number it invented, or attributing a claim to a document it never actually read. This can slip past both trajectory eval (the tool calls were correct) and goal-success eval (the user got an answer that satisfied them) while still being a serious quality and trust problem. Build a specific check: for every claim in the agent's final response, is it traceable to something the agent actually retrieved?

Safety and harmlessness evals are a separate suite, not a subset of capability eval. Whether the agent can complete a task well and whether the agent behaves safely under adversarial or edge-case pressure — prompt injection attempts, requests to exceed its scope, ambiguous instructions with a harmful interpretation — are different questions requiring different eval design, usually adversarially constructed rather than sampled from typical production traffic. Don't let safety evaluation ride along as an afterthought inside your general capability eval suite; it needs deliberate, adversarial test construction of its own.

Cost and latency belong in the eval report, not just a separate ops dashboard. A model swap that improves task success by 2% but doubles token cost and triples p95 latency is not an unambiguous win — it's a tradeoff that needs to be visible in the same report as the accuracy numbers, evaluated by the same people making the ship decision, at the same time.

Eval sets go stale — build a refresh cadence, not a one-time asset. A static eval set, reused unchanged for months, gradually becomes disconnected from actual production usage as user behavior shifts, and can become an implicit overfitting target if the same set is used repeatedly through many iteration cycles. Treat the eval dataset as a living asset: resample from production periodically, retire scenarios that no longer reflect real usage, and add new scenarios as new task types emerge.

Offline eval and online eval are complementary, not substitutes for each other. Everything above describes offline evaluation — testing against a fixed or semi-dynamic dataset before shipping. You also need online evaluation: shadow-mode deployment where a new model runs alongside the production model on live traffic without affecting the user, and canary releases where a small percentage of real traffic sees the new model with close monitoring. Offline eval catches most regressions before they reach users. Online eval catches the regressions that only manifest in the full complexity of real production traffic, which no offline dataset — however well constructed — fully captures.

⚡ The practitioner take

The teams that get burned by agent evaluation are almost never the teams that skipped evaluation entirely — everyone knows to build some eval set before shipping. They're the teams that built an eval set that looked rigorous, produced a clean single number, and gave false confidence, because it graded only final outputs, ran each scenario once, wasn't sized for statistical power, and drifted out of sync with what production traffic actually looked like months after it was built.

Rigorous agent evaluation is genuinely harder than model evaluation was, because agents introduce trajectory, tool use, multi-turn dynamics, and stochasticity as first-class problems rather than edge cases. The investment required — dynamic branching datasets, judge calibration, statistical power analysis, slice-level reporting — is significant. It is also the only thing that makes it possible to change your model with confidence instead of hope, which given how fast model releases are currently shipping, is not optional infrastructure. It's the thing that determines whether you can move fast at all.

Build the eval suite you'd need to trust before your CEO asks why the new model version broke something in production that the eval never caught. That's the bar.

— Santosh

👀 Also Watching

If someone forwarded this to you, subscribe at whattheagent.com. If this was useful, forward it to one engineer who needs it.