Part III · Survive Monday / Chapter 9
Observability, Evaluation, and Debugging
Trace the whole path. When quality drops, the model is usually not the culprit.
At 8 a.m. the agent quoted the wrong statute, and a customer noticed. You open the dashboard: traffic healthy, latency fine, no errors. The agent didn’t crash. It just lied — and nothing you were watching was built to see it.
Chapter 8 got your system deployed, running, and bounded. This chapter answers the question that decides whether it survives contact with reality: how do you know it’s working? And when it isn’t, how do you find out why? This is the discipline the demo-builders skip and the survivors obsess over. It is, more than any other, the chapter that separates the five percent who realize value from the ninety-five percent who ship something they cannot diagnose.
The gap traditional monitoring can’t see
You already know how to watch software in production. Latency, uptime, error rates, throughput, resource usage — the dashboards you’ve trusted for years. Point them at an AI system and they will tell you, cheerfully and correctly, that everything is fine while your model hands a customer a confident fabrication. A model can return a 200 response in fifty milliseconds and still hallucinate, leak private data, or produce biased output. By your monitoring’s definition, nothing was wrong. By any definition that matters, everything was.
That gap is the reason this chapter exists, and closing it means keeping three things distinct that get carelessly lumped together. Monitoring watches — is the system up, is it fast? Observability traces — what actually happened inside this one request? Evaluation judges — was the output any good? You need all three, but the new and genuinely hard one, the one none of your existing tools do, is evaluation. Judging quality is the layer AI forces you to add.
And notice how this completes the arc from Chapter 8. There, control came from bounding what the system can do, because you can’t make the model deterministic. Here is the other half: you also can’t make it predictable, but you can make it fully visible and measurable. You may not know in advance what any given output will be, but you can see every one, score every one, and trace every one to its cause. That is a different kind of control than you’re used to — not prediction, but total observability after the fact — and for a non-deterministic system it is exactly the kind you can actually have.
The trace tree is the unit
Before you can judge quality, you have to see what produced it — and a single AI request is almost never a single call. It’s a tree: the model generation, the retrieval that fed it, the tools it invoked, the steps an agent took along the way. That tree is the response path from Chapter 8, and observing the whole thing, span by span, is the foundation everything else in this chapter stands on.
The architecture the field converged on in 2026 is worth stating precisely, because getting it right protects you and getting it wrong locks you in. LLM observability is OpenTelemetry tracing, plus LLM-aware span conventions (the OpenTelemetry-GenAI and OpenInference standards) that name the things that matter — the model, the prompt, the completion, the tokens, the tool call — plus evaluation scores written back onto those spans. The trace tree is the unit of work, and each span carries its own inputs, outputs, cost, and, once scored, its own quality verdict.
The reason to insist on the open standard is the reason this book insists on it everywhere: instrument once against OpenTelemetry and your traces flow to any backend you choose, and can be swapped later with a config change; adopt a vendor’s proprietary span format and every future choice locks tighter around it. Neutrality belongs at the observability layer as much as at the model layer. What this buys you, concretely, is per-step cost and token attribution (the FinOps thread from Chapters 5 and 8, picked up again in Chapter 10) and — more importantly here — the raw material for both evaluation and debugging. You cannot judge or fix what you cannot see. The trace is how you see.
Evaluation: judging quality you can’t unit-test
Here is the problem in one sentence: you cannot write
assert output == expected for open-ended, non-deterministic
text. So how do you judge quality, at scale, without a human reading
everything? Three families of method, used together, from cheapest to
most nuanced.
Programmatic checks come first because they’re free and objective: does the output parse, does it match the required schema (Chapter 6’s structured outputs), does it contain the citation it was supposed to? Similarity and entailment methods go further: embedding similarity to a reference answer, or semantic entailment against the retrieved context to check groundedness. And LLM-as-a-judge handles the subjective core — a model scoring another model’s output against a rubric for factuality, relevance, faithfulness, or tone. This last one is now the workhorse of AI evaluation, and it comes with a caution you must respect: a model judging a model has blind spots of its own, and two judges will often fixate on different evidence and reach different verdicts. So you align your judge against human ratings on a sample before you trust it, you keep a human in the loop where the stakes are high, and you treat an unvalidated judge score as the unreliable number it is. A judge you haven’t checked against people is not a measurement; it’s a guess with a decimal point.
The metrics worth scoring are the ones the research has grounded: faithfulness and groundedness (does the answer stay true to its source?), hallucination, relevance, toxicity, bias, citation support — and, for retrieval systems, the four signals from Chapter 7: retrieval recall, context relevance, answer faithfulness, and answer correctness. Against these you run a golden dataset — a curated set of representative inputs with known-good criteria, your standing yardstick for quality.
But the single practice that most separates the five percent from the ninety-five is not the golden set itself. It’s the feedback loop around it: production traces flow to human annotation, where domain experts, product managers, and QA review and label real failures, and those labeled failures are curated back into the golden dataset for the next cycle. Your test coverage evolves with real usage, so it keeps catching what actually breaks in the wild rather than what you imagined might break at the start. And this is a team sport, not an engineering chore — the people who know what “good” means for a legal answer or a clinical summary are the domain experts, not the engineers. Build the annotation workflow so they can contribute without engineering as the bottleneck, because that is where quality actually gets defined. A team that treats evaluation as one person’s side task has already decided to be in the ninety-five percent.
Three places evaluation lives
The same evaluation discipline shows up at three points across the lifecycle, and a serious system uses all three.
Pre-deployment is the gate from Chapter 8: run the golden set before a change ships, and catch the regression — a prompt rewrite that quietly drops factuality from ninety-four percent to eighty-nine — before a single user meets it. Production monitoring scores live or sampled traffic after deployment; because real volume dwarfs any test set, you lean on cheaper scorers and sampling, and this is how you catch drift. Runtime guardrails inspect the output before it ever reaches the user — blocking, rewriting, or routing a high-risk response for review, and intercepting a dangerous agent action before it executes, a leaked identifier or a policy violation stopped in flight. This last one is where evaluation meets Chapter 8’s control frame and Chapter 11’s security: it’s the real-time safety net beneath everything.
Same discipline, three touchpoints: catch it before you ship, watch it after you ship, and stop the worst of it in the moment.
Catching drift: the degradation nobody deployed
Chapter 8 named the fear; this is where you detect it. Silent drift is a gradual decay in quality — or cost — that no aggregate dashboard reveals, because averages smooth it over and latency stays perfectly flat while the answers quietly get worse. Nobody deployed the degradation. It crept in.
Trace it back and every cause lives somewhere in the response path. A retrieval index went stale, so the model grounds on last quarter’s facts. The provider silently updated the base model — Chapter 8’s prompt drift — and your unchanged prompt now behaves differently. A prompt edit reached production without enough evaluation coverage. The embedding distribution shifted underneath your vector store. In each case your code never moved, and the quality fell anyway.
Detecting it requires one inversion of your instincts: alert on quality, not just on latency. The alarm should fire when an evaluation score drops, not only when a request is slow — and it should watch drift at the level of the individual prompt, use case, or user segment, not just the global aggregate, because a serious problem in one workflow drowns unnoticed in an average across all of them. The goal is blunt and it is serious: know before your users do, and in a regulated industry, know before your regulators do.
Debugging non-determinism
When quality does drop, you face the last unfamiliar problem: how do you debug something that never behaves the same way twice? Not the way you debug deterministic code. There is no single broken line to find. Instead you reason about a path and a distribution, and the trace tree is what makes that tractable.
The method is root-cause by elimination, walking the tree. Was it retrieval — wrong chunks, or none (Chapter 7)? The prompt — a recent edit? The model — drift, or an upstream update? A tool — an error the agent quietly worked around? The router or a fallback — a different, weaker path served without anyone noticing? Carry Chapter 8’s lesson with you as your prior: the model is usually not the culprit, and the trace is what tells you which component actually moved. Debugging here is detective work over a recorded path, not a breakpoint in a repeatable run.
And this is where the thread that started in Chapter 1 finally ties off. Non-determinism was named early as the one genuinely new muscle, and it has shadowed every chapter since. Here is where it stops being frightening. You were never going to get a deterministic model, and chasing one was the wrong goal all along. What you can have — what this chapter builds — is a system that is completely observable, measurable, and diagnosable: every output seen, every output scored, every output traceable to its cause, even though none of them can be predicted in advance. That is control of a different kind than deterministic software gave you, and for this material it is not a lesser kind. It is the right one.
Observability, evaluation, and debugging are the least glamorous work in the book, and the most decisive. They are what turn a demo that dazzled in a conference room into a system you can trust on Monday and defend to an auditor on Friday. They are precisely the discipline the ninety-five percent skip on their way to a launch that quietly rots. Building this is what puts you in the five percent — and, just as importantly, what keeps you there.
Experiments
Four experiments in seeing what your old logs never showed you.
- Trace one real request end to end. Instrument a single request and look at the whole tree — generation, retrieval, tool calls, each with its inputs, outputs, and cost. Notice how much your old logs never showed you.
- Score a golden set. Build a small set of representative inputs, pick one metric — faithfulness is a good first — and score your outputs to get a real number. Then curate five genuine failures back into the set. You’ve just started the feedback loop.
- Alert on quality, not latency. Wire an alert that fires when an evaluation score drops rather than when a request is slow. Break something on purpose and watch it catch what your infrastructure dashboards never would.
- Root-cause a bad answer. Take one wrong output and walk its trace to name the actual culprit. Bet, before you look, that it’s the model — and notice how often it isn’t.