Deploying and Operating AI Workloads

Getting an AI feature to work in a notebook and getting it to survive Monday are different problems. The second one is mostly about staying in control of something that, by design, you cannot fully predict.

You’ve built something — a grounded assistant, a workflow, maybe an agent. This chapter is about the moment it stops being yours to watch and starts running on its own: deployed, unattended, changing over time, and — if it can act — capable of doing real things in the world. That transition is where most AI dies, and it dies not from weak models but from the absence of the operating discipline this chapter is about.

The discipline has a name, LLMOps, and it ends somewhere the earlier chapters have only hinted at: control. Because the thing you’re deploying is non-deterministic by design, and once it can act, the difference between a useful system and a dangerous one is entirely in how well you’ve bounded it. So we build up to that honestly — and then defuse the fear, because staying in control turns out to be ordinary engineering.

The notebook-to-Monday gap

Getting an agent to run in a notebook while you watch is a fundamentally different problem from getting it to run reliably, unattended, at scale. The gap between those two is this chapter, and it starts with a shift in where your attention goes.

Classic machine-learning operations is built around one object: the model. You train it, version it, deploy it, watch its predictions for drift, and retrain when it degrades. Operating an LLM application inverts this. You almost never train the model — you consume a pre-trained one through an API — so the model is often the least-frequently-changed component in the whole system. What changes constantly, and what you actually have to operate, is everything around it: the prompts, the retrieval configuration, the choice of model and provider, the tool definitions, the evaluation thresholds. The center of gravity moves from “managing a model” to “operating a product whose behavior changes.”

Structurally, none of this is new to you. It’s the software lifecycle from Chapter 4 — source control, CI/CD, infrastructure-as-code, environments promoted from development through staging to production — specialized for AI. What’s new is what those pipelines now carry, and what has to be true before a change is allowed through the gates between environments. That’s the rest of the chapter. And the reframe to hold onto from the start: deploying is not the finish line of building. It’s the beginning of operating — and operating a system that can surprise you is, at its core, a control problem.

Prompts are deployments; evals are the gate

Start with the smallest unit of change, because it carries the biggest surprise. A prompt is code. A one-word change to a system prompt can move output quality more than a full model retrain would in classic machine learning. So every prompt change — and every retrieval-config or model-selection change — is a deployment: versioned, reviewed, tested, and reversible, treated with exactly the seriousness you’d give a code change, because it is one.

But how do you test a change whose output is non-deterministic? You can’t write the unit test “assert the answer equals four” when the answer legitimately varies run to run. This is the shift that catches teams: evaluations replace unit tests. You run the proposed change against a golden dataset — a curated set of representative inputs with known-good criteria — and gate the deployment on the aggregate result. The change merges only if it clears the eval bar. Your pipeline stops asking “does this function return the right value” and starts asking “does this change make the system’s behavior better or worse across cases that matter.”

That gate has a real cost: it means your continuous-integration pipeline has to actually call the model on every prompt change — slower and more expensive than a unit test that runs in milliseconds. The pattern that keeps this affordable is layered gates. Cheap, fast, free checks first: does the output parse against its schema (the structured outputs of Chapter 6)? Then the expensive behavioral evals, the ones that spend real tokens. Then, where the stakes justify it, a human review queue. Fail fast and cheap, and spend the expensive gate only on what already passed the free one.

Underneath sits a registry: prompts, model selections, and configurations versioned in the repository, alongside code, deployed together. At any moment you can see which prompt version is live and roll back to a previous one. (Building a good golden dataset, and the deeper evaluation methods, are Chapter 9’s job. Here, evaluation is simply the gate that stands between a change and production.)

The artifact is the whole response path

Here’s a trap that sends teams debugging in the wrong direction. Ask “what version is running?” and the instinct is to answer with the model. But the model is a sliver of what actually produced any given output.

To reproduce, debug, or roll back a response, you need the entire response path: the prompt version, the model and its generation parameters, the retrieval configuration and what it actually fetched, the tool set and what the tools returned, and which rollout segment served the request. That whole path — not the model name — is the artifact you version and observe.

Why it matters is the everyday reality of operating these systems: when quality degrades, the model is usually not the cause. A prompt template changed. The document index updated overnight. The router sent this request to a different model than yesterday. A tool returned an error the agent quietly worked around. The fallback route fired and served a cheaper, less-validated result. A team that captures only “the model was X” sees a fraction of the system and debugs blind, chasing the one component that probably didn’t move. Operate the path, not the model — and you turn “something changed and quality dropped, no idea what” into “we can point at the exact change.” That capability is what Chapter 9’s tracing makes real.

Shipping non-determinism safely

Two challenges are unique to shipping a system whose outputs you can’t predict exactly, and both have well-worn answers borrowed from ordinary distributed systems.

The first is rollout. Because you can’t prove an output correct the way you prove a function returns four, you don’t flip a switch — you release gradually and compare. Canary: send the change to a small slice of traffic and watch before widening. Shadow: run the new version alongside the old on real traffic without showing its output to users, and compare quality offline — the safest way to catch a regression before anyone experiences it. Champion/challenger: keep the current version as the champion and test challengers against it on live cases. Fallback routing: when the primary path fails or misbehaves, drop to a known-good one — the gateway from Chapter 3 earning its keep again. And under all of it, a rollback path: any change reverts instantly, precisely because you versioned the whole response path.

The second is stranger, and it’s the one that surprises even careful teams: prompt drift, the degradation you didn’t cause. Your provider quietly updates the base model, and your unchanged prompt starts producing worse output overnight. Your code didn’t move. Your dashboards are green. And your users are getting confidently wrong answers. This is the non-deterministic world’s version of “but it worked yesterday,” and it can arrive while you sleep. The defenses are architectural: pin or version the model where the provider allows it; shadow-test model upgrades before they go live; and keep the gateway in place so you can pin, swap, or roll back the model like any other dependency. (Detecting drift in production belongs to Chapter 9; the point here is to build so that when it happens, you can respond.)

The theme is now unmistakable. You are operating something that changes under you and can surprise you. That is not a defect to engineer away — it’s the nature of the material. Which brings us to the two hardest expressions of it: agents, and control.

Deploying agents: long-running distributed systems

Chapters 6 and 7 handed the operational reality of agents forward to here, and it comes down to a reframe that should feel reassuring rather than daunting. An agent in production is a long-running, multi-step process that spans several models, tools, and services and holds state over minutes, hours, or days. Strip away the AI vocabulary and that is a distributed system — which, as a cloud-native engineer, is exactly the thing you already know how to run. The AI doesn’t invent new problems here so much as supercharge familiar ones.

The questions that matter appear only after you deploy, which is why notebook agents don’t prepare you for them. What happens when the agent crashes on step seven of ten? When a tool times out mid-run? When the process restarts? Ignore these and a single late-stage failure re-runs the entire workflow from the beginning — re-paying, in tokens and latency, for every model call it already made. The answer is durable execution: state persistence, automatic retry, and crash recovery, so a failure resumes from where it stopped instead of starting over. An honest note worth having: the popular agent frameworks often don’t provide this natively — you bolt on durable-execution infrastructure. That’s not a failing; it’s recognizable reliability engineering, the same you’d apply to any long-running workflow.

The rest is orchestration: routing between steps, managing state, handling errors, and inserting human checkpoints where they matter — observed through open instrumentation (the OpenTelemetry of Chapter 4) so you get one coherent view of a multi-step, multi-system flow, including where the tokens are going at each step. This is the “harness” from Chapter 6 made concrete, and the maxim holds: agents aren’t hard; the harness is hard. Route what you can to typed, testable code instead of letting the model decide everything; abstract model calls behind versioned interfaces so you can pin and swap; treat cost per workflow step as a first-class metric. The stakes are not hypothetical — only a small fraction of enterprise agent pilots have reached production at scale — and the gap between the pilots and the survivors is almost entirely this operational discipline, not model capability.

Staying in control

Now the part that has to be said plainly, because it’s the genuinely uncomfortable one — and because you deserve the honest version followed by the way through, not reassurance that skips the hard bit.

Everything in this chapter has circled a single fact. You are deploying a system that is non-deterministic — the same input can yield different actions. It can change under you, through drift and silent model updates. And once it’s an agent with tools, it can act in the real world: spend money, send messages, alter records, reach external systems. Non-deterministic, plus changeable, plus able to act, is a different risk profile from any software you have shipped before, and it would be dishonest to pretend otherwise. NIST’s 2026 work on agents names precisely these traits — autonomous real-world actions, runtime tool-switching that defeats static rules, memory that can be poisoned over time, and non-deterministic behavior — as the things existing frameworks were never built to handle. This is the scary part. Sit with it for a moment, because the way out depends on taking it seriously.

Here is the reframe that dissolves the fear, and it comes from NIST rather than from optimism. Perfect control of the model is provably impossible. A 2026 peer-reviewed result, extending Gödel’s incompleteness theorems, established that no finite set of guardrails can be universally robust against adversarial input — and because a model reads instructions and data through the very same channel, there is no clean way to separate trusted commands from untrusted content. If your plan was to make the model itself safe and predictable, that plan cannot work, for you or for anyone. So you stop trying to, and the anxiety attached to that impossible goal goes with it. Control does not come from perfecting the model. It comes from bounding what the system can do. You assume the model can misbehave or be manipulated, and you engineer so that when it does, the damage is contained. The goal, as the security community frames it, is not to stop the agent from acting — it’s to ensure it only acts as intended, and to shrink the blast radius when it doesn’t, so that a single failure stays a single failure instead of becoming a system-wide disaster.

That reframe turns an unbounded fear into a bounded engineering task, and the task is one you already know how to do. You wrap the one probabilistic component — the model — in deterministic controls, every one of them ordinary:

Least privilege. The agent gets a scoped identity and default-deny access to tools and data, with short-lived, narrowly-scoped credentials. It can touch only what it must. This is the single highest-leverage control, because it directly caps the blast radius no matter what the model decides.
Human-in-the-loop on the irreversible. Any high-stakes or irreversible action — moving money, deleting data, sending an external message — passes through a human checkpoint. Regulation increasingly requires exactly this for high-risk systems (Chapter 11), but you’d want it regardless.
Sandboxed execution. Tools and code run confined, where a bad call can’t reach past its box.
Bounded execution. Hard limits on steps, loops, time, and spend, so a confused agent can’t run away — no infinite loops, no runaway bill.
Kill switches. A stop you can hit at runtime, and that the system can trip itself on anomaly.
Observability. You cannot control what you cannot see; the trace and the audit trail (Chapter 9) are part of the control surface, not a separate concern.

The organizing principle that keeps all this proportionate rather than paranoid is simple: match the control to the blast radius. An agent that drafts text a human will review needs a light touch. An agent that can execute trades or modify production data needs the full stack — tight permissions, human approval, sandboxing, hard limits, a kill switch. Size the harness to what a failure could actually cost. And there is a calm fallback, drawn straight from the agent-security maturity guidance, for when you can’t get comfortable: if you cannot control an agent enough for the power you were going to give it, you have two honest choices — invest in stronger controls, or reduce the agent’s power. Either one keeps you in control. You are never cornered into choosing between “unleash it” and “don’t build it.”

None of this is AI magic. Containment, least privilege, blast-radius thinking, circuit breakers, fail-safes — this is distributed-systems and security engineering, the discipline you have practiced for years, pointed at a new kind of component. The model is the single part you can’t make deterministic; everything you build around it is as controllable as it has always been. That is the whole frame, and it’s why there’s no need to panic: the goal was never a deterministic model, which is impossible. The goal is a controlled system, which is entirely achievable — and mostly with tools already in your hands.

So you operate an AI system the way you’d operate anything powerful and changeable: with gradual rollout, instant reversibility, a record of who approved which change on the basis of which checks (Chapter 4’s shift-left governance, now living at runtime), and controls sized to the stakes. Deploying was never the finish line. It’s the beginning of staying in control of something that earns its keep precisely because it can do things you didn’t script — safely, because you decided in advance what “things” it is allowed to do.