How Models Are Served and Consumed

The model feels like magic until the invoice arrives. Then it feels like a metered utility — which is exactly what it is, and exactly what makes it something you can engineer.

Chapter 4 got you to a running model behind a control plane. This chapter is about what actually happens each time you call it — because until you can see the meter, the clock, and the ceiling, every AI feature is a gamble on cost, speed, and quality that you won’t understand until it’s in production and one of the three has gone wrong.

The good news is that all three are legible. A model is not an oracle; it’s a service with predictable economics and physics. Learn to read them and you can look at a proposed feature and predict, before you build it, roughly what it will cost, how fast it will respond, and where its quality will quietly break. That predictive power is the difference between the teams that ship AI that survives and the ones whose pilots die on contact with a real bill and a real user.

---

The unit of everything: the token

Everything downstream is denominated in one unit, so start there. A token is a chunk of text the model processes as a single step — very roughly four characters, or three-quarters of a word, for English prose. Code tokenizes more densely, often half again as many tokens per line, because of all the punctuation and structure. And every provider uses a slightly different tokenizer, so the identical paragraph can cost a different number of tokens on different models — a subtlety that quietly distorts naive cross-provider price comparisons.

Two kinds of token get billed, separately. Input tokens are everything you send: the prompt, the instructions, the retrieved documents, the conversation history. Output tokens are everything the model generates back. And here is the fact to burn into memory, because half of cost engineering follows from it: output tokens cost several times more than input tokens — commonly somewhere in the range of two to eight times more — across essentially every provider.

That asymmetry isn’t a pricing whim. It’s physics. Your input can be processed in a single parallel pass — the model reads the whole prompt at once, a phase called prefill. Output cannot: each generated token requires its own full pass through the model, produced one at a time in sequence, a phase called decode. Generating is inherently more expensive than reading, and the price card simply reflects that.

The immediate consequence is that the shape of your workload sets its cost. A summarization task — long input, short output — is cheap per call. A generation task — short input, long output — is expensive, even at the same total token count. Which means the single highest-leverage cost decision you make is almost never which provider to negotiate with. It’s how many output tokens you let the model produce. Keep that in your pocket; we’ll come back to it.

The bill is a design decision

Sticker prices span an enormous range — a couple of orders of magnitude between the cheapest small model and the priciest frontier reasoning model — and they keep falling fast, as Chapter 1 showed. But the sticker price is a ceiling, not what a well-built system actually pays. Three levers sit under it, and they stack.

Route by difficulty. Most of what an application does — classifying, extracting fields, formatting, routing, first drafts — does not need the smartest model in the world. Send that work to a cheap, fast, small model and reserve the expensive frontier models for the genuinely hard requests. Done well, this alone cuts spend by more than half, often much more. This is the same multi-model posture Chapter 3 built the gateway for; here it shows up as a line on the invoice.

Cache the repeated parts. Most prompts have a stable prefix that repeats on every call — the system prompt, the few-shot examples, a reference document. Providers can cache that prefix server-side and bill it at a steep discount, commonly well over half off, sometimes up to ninety percent off the cached portion. There’s a second, quieter benefit: cached reads frequently don’t count against your rate limits, so caching buys you throughput as well as savings. The catch is architectural — you only benefit if your prompts are stable and front-loaded, which is a design choice you make on purpose.

Batch what isn’t urgent. For anything that doesn’t need an answer this second — nightly reports, bulk classification, data enrichment — providers offer asynchronous batch processing within a window, typically at half price. This is close to free money, and most teams leave it on the table.

Stack caching and batching on routed traffic and your effective cost can fall to roughly a quarter of the on-demand rate. None of this is exotic; it’s just deciding to do it.

One trap to name, because it’s where teams fool themselves: cost per token is not cost per task. The cheapest model per token can be the most expensive per job, if it needs more retries, produces worse output that has to be regenerated, or forces you to stuff in more context. The number that matters to the business is cost per unit of work — per transaction, per resolved ticket, per document processed. And here’s the uncomfortable statistic: only about a fifth of organizations actually track their AI spend at that granularity. The rest are flying blind, which is why so many pilots die when someone finally does the division. Instrument cost per unit of work from the first day — it ties directly into the governance of Chapter 4 and the observability of Chapter 9.

And never forget the lever from the last section: shorter output is cheaper output. Asking for concise, structured responses — a JSON object instead of a page of prose — routinely cuts the expensive half of the bill without touching quality.

The reasoning tax

There’s a newer cost dynamic that breaks a comfortable assumption, and you need to see it coming.

Reasoning models don’t just answer. They first generate a long, hidden chain of “thinking” — working through the problem, testing approaches, backtracking — and only then produce the visible response. Those thinking tokens are real tokens, billed as output tokens, the expensive kind. And they are not cheap in number: for hard problems, a reasoning model can consume roughly an order of magnitude more tokens per useful answer than a plain model.

That imposes two taxes, not one. A cost tax, because you’re paying output rates for a large invisible preamble. And a latency tax, because thinking tokens are autoregressive and happen before the first visible word — so a heavy reasoning chain can mean seconds of silence before the user sees anything at all. Time-to-first-token, which we’re about to make central, balloons.

This is why the industry’s favorite reassurance — “cost per token keeps falling, so this all gets cheaper” — is only half true. Per-token prices do fall. But if you shift your workloads toward heavy reasoning, tokens-per-answer step up by that order of magnitude, and per-answer cost can rise even as per-token cost drops. The two trends fight, and in a reasoning-heavy system the reasoning wins.

The discipline, then, is to treat reasoning as a governed resource rather than a free upgrade. It’s usually exposed as a dial — off, low, medium, high — and, importantly, its returns diminish: past a certain depth, more thinking stops improving the answer, and what actually predicts quality is the tokens genuinely consumed, not the budget you allocated. So do with reasoning exactly what you did with model choice: route by difficulty. Default the bulk of traffic to no or low reasoning, and spend the deep-thinking budget only on the requests that provably earn it. A system that makes every request think is a system quietly overpaying in both money and time.

The clock: latency as a first-class constraint

Speed is not one number, and conflating its two components is a common source of “the model feels slow” confusion.

The first number is time-to-first-token — how long the user waits before anything appears. The second is throughput, or time-per-output-token — how fast the response completes once it has started. They come from the two phases we already met: prefill (reading your input) dominates time-to-first-token, and decode (generating output) dominates throughput. So a very long input makes the user wait longer to see the first word, while a very long output makes the whole response take longer to finish. Two different levers on two different delays.

This is why streaming exists, and why it’s a product decision more than a technical one. By delivering tokens as they’re generated rather than waiting for the whole response, streaming makes the perceived latency equal to time-to-first-token instead of total time. A ten-second full response becomes a one-second wait that then keeps flowing — which, for anything interactive, is the difference between “broken” and “fine.” For a chat or an assistant, streaming is close to mandatory; for a background job, it’s irrelevant.

Two working rules. First, the datasheet speed is marketing: real latency varies with load, region, model, and time of day, so measure on your own traffic at your own load before you promise anyone a number. Second, latency is an architecture decision that follows from the product. Interactive features need low time-to-first-token and streaming; background and bulk work can tolerate seconds or hours — and should therefore ride the batch discount from earlier. Match the serving mode to the latency budget the feature actually has, rather than paying real-time prices for work no human is waiting on.

The ceiling that isn’t where it says: context

Every model advertises a maximum context window — the number of input tokens it will accept — and those numbers have grown spectacularly, now reaching a million tokens and beyond. It is tempting to read that as “I can put everything in and let the model sort it out.” Resist that, because the advertised window and the usable window are not the same number, and the gap is where a lot of production quality quietly dies.

The advertised window is the maximum the API accepts. The effective context — how much the model can actually reason over without losing precision — is often dramatically smaller, and quality starts degrading well before the stated limit. Two well-documented effects drive this. The first is lost in the middle: models attend well to the beginning and end of a long input and poorly to the material in between, a U-shaped curve that has been measured at accuracy drops of thirty percent or more for information buried in the center. The second is broader and more unsettling — context rot. Research that stress-tested eighteen different frontier models found that every one of them got worse as the input grew longer, even when the window was nowhere near full. More tokens in, worse output out. The mechanisms compound: lost-in-the-middle, plus attention spreading ever thinner as the token count rises, plus irrelevant-but-similar content actively distracting the model.

Notice the two-sided penalty here, because it’s unusually clean. Stuffing the context window costs you money (all those input tokens) and costs you quality (rot). Context discipline is the rare lever that is simultaneously the cheaper choice and the better one — there is no trade-off to agonize over.

And the failure mode you should fear most is the silent one. Overrunning the hard limit throws a loud, obvious error; you’ll notice immediately. Context rot throws nothing. Your code runs, your logs are clean, and the answers just quietly get less reliable as the input grows — the most expensive kind of bug, the kind nobody sees until a user does. So the operating rules are: put your most critical instructions and data at the edges of the prompt, not the middle; don’t dump everything in when you can retrieve only what’s relevant — which is the entire argument for the retrieval systems in Chapter 7; and smoke-test your own data before trusting anything past the lower portion of an advertised window. Big windows are table stakes now. They are not a substitute for deciding, deliberately, what belongs in the prompt.

Serving shapes, and the self-host crossover

Pull the threads together and the ways you consume a model resolve into three shapes, each matched to a different latency budget. Real-time synchronous calls for interactive work, at full price. Streaming for interactive work where perceived speed matters, which is most of it. And batch for volume and background work that can wait, at roughly half the cost. Choosing the shape is choosing where a workload sits on the speed-versus-cost line, and most systems use all three for different jobs.

Two constraints shape the choice in practice. Rate limits and quotas are not a footnote — the cheapest model on paper is useless if you can’t get enough throughput at your spend tier, and this can override a per-token comparison entirely. It’s worth knowing that caching and batch often carry separate, higher limits, so they buy you capacity as well as savings; design for limits with backoff and queueing rather than discovering them in an incident. And the self-host crossover from Chapter 4 is fundamentally an economic line: consuming a managed endpoint is the right call until your volume and utilization are high enough that owning the accelerator is cheaper — a threshold that lands, very roughly, in the range of tens of millions of tokens a month at sustained high utilization, and only when you have the operational capability to run it. Below that line, managed wins on every axis; above it, self-hosting can pay. The gateway from Chapter 3 is precisely what lets you cross that line later without rewriting the application.

Beyond text: multimodal inputs

This chapter has assumed text in and text out, as most of this book does for clarity. But real enterprise data is not tidy text — it’s PDFs, scans, invoices, contracts, charts, screenshots, diagrams, forms, product photos, and call recordings. Increasingly the models take all of it directly, and the good news is that you already hold the mental model for it.

The mechanism, in one line: everything becomes tokens. A multimodal model runs an image through a vision encoder that turns it into vision tokens, maps those into the same space the text tokens live in, and processes the whole mix in a single attention pass; audio is handled the same way, either tokenized directly or transcribed to text first. So every mechanic in this chapter — the token as the unit, the context window, the cost math, the latency clock — extends directly to images and audio. You simply have new sources of tokens.

The cost twist is the part you must not miss: images are token-hungry. A single high-detail image can cost anywhere from a few hundred to many thousands of tokens depending on the model and its resolution, enough that bolting vision onto a text workload can multiply its bill several-fold overnight. So the levers from this chapter grow multimodal-specific dials. Image resolution and “detail level” become cost levers — send a low-detail version when a thumbnail would answer the question; audio gets chunked; and video, billed per second, is the most expensive input of all. Curiously, the arrow sometimes points the other way: for a dense document, handing the model the page as an image can cost fewer tokens than its extracted text, which is why compressing documents into images is a live optimization thread.

And the jagged frontier of Chapter 2 reappears, now sliced by modality. Multimodal models are genuinely impressive and predictably uneven — they reason beautifully across a chart or a screenshot or a page, then get beaten on narrow, high-accuracy extraction by purpose-built tools: a dedicated OCR service for pulling exact text off a messy scan, a dedicated speech model for transcribing accented or noisy multi-speaker audio. So the mature pattern is not “use the multimodal model for everything.” It’s the tool-use pattern of Chapter 6 applied to perception: let a specialized extractor do the precise, high-stakes reading, then hand its clean output to the model for reasoning. Route by what each part actually needs, exactly as you route text tasks by difficulty.

For the enterprise this lands first on document understanding — invoices, contracts, forms turned into extracted, structured fields — which plugs straight into the retrieval of Chapter 7 and the agents of Chapter 8, where a model that can see a screen, read a document, and act is the substrate the agentic future is being built on. Carry one caution forward: an instruction can hide inside an image as easily as inside text, so multimodal widens the prompt-injection surface of Chapter 11. Treat pixels as untrusted input, too. Multimodal is not a separate discipline; it is this chapter’s discipline with new token sources and a bigger bill.

Step back and the whole chapter collapses into one idea. Model choice, serving shape, caching, batching, reasoning effort, output length, and context discipline are not separate concerns — they are the dials on a single console. Engineering inference means setting those dials to hit the cost, latency, and quality budget each workload actually has, and then instrumenting the system well enough to see, in production, whether you got it right. That last part — seeing whether you got it right — is a whole discipline of its own, and it’s where we’re headed.

---