Data, Retrieval, and Grounding

A model with a million-token window and no discipline will still confidently tell you the wrong thing. Grounding is how you make it tell you the right one — and prove where the answer came from.

Chapter 6 established grounding as a pattern: retrieve the relevant information, then generate. This chapter is the mechanics — how you actually turn a pile of enterprise documents into something a model can answer from accurately, safely, and verifiably. It matters more than any other build chapter, because grounding is where most genuinely useful enterprise AI lives. The value is almost always in your data — your policies, your contracts, your tickets, your history — and no model was trained on that.

It opens by clearing away the single most common piece of bad advice you’ll hear this year.

RAG didn’t die; it specialized

Every few months someone declares that retrieval is dead. The argument goes: context windows are a million tokens now, so stop building retrieval pipelines and just put everything in the prompt. It sounds reasonable. It’s wrong, and Chapter 5 already told you why — the effective window is far smaller than the advertised one, quality rots as you fill it, and stuffing context is expensive. But you don’t have to take the earlier chapter’s word for it. Independent benchmarking that pitted retrieval against long-context across thousands of cases concluded that neither is a silver bullet: retrieval wins when the corpus is large, when freshness matters, and when you need to cite sources — and long-context runs something like eight to eighty times more expensive at scale.

So the real question was never “RAG or long context.” It’s which tool fits which query. Whole-document reasoning — “summarize what we discussed” — suits a long window. A precise fact that must be exactly right and cited — “quote the termination clause” — suits retrieval. Style and formatting consistency isn’t a grounding problem at all; that’s fine-tuning. Questions that require connecting facts across sources call for the graph methods later in this chapter. The best teams don’t argue the dichotomy; they route by query type, then measure.

What’s actually happening is that retrieval is specializing and disappearing into the infrastructure — becoming the invisible “context layer” the way a database is the invisible layer beneath an ordinary app. Nobody asks “should this app use a database”; soon nobody will ask “should this system use retrieval.” The question becomes how is our context layer configured — which is Chapter 6’s context engineering, made operational. The market agrees: this is a category growing at roughly fifty percent a year. Dead technology doesn’t scale like that.

How text becomes retrievable: embeddings and chunking

To retrieve by meaning, you first turn text into numbers. An embedding is a vector — a point in a high-dimensional space — positioned so that texts with similar meaning land near each other. Search then becomes geometry: find the points nearest to the query. This is what lets a search for “how do I expense a flight” find a document titled “travel reimbursement policy,” despite no shared words.

But hold one fact about embeddings close, because half of this chapter follows from it: they are lossy by design. Compressing a whole paragraph into a single point necessarily throws information away. That compression is what makes semantic search possible, and it’s also why semantic search alone is not enough — a point in space can’t preserve every exact term the paragraph contained.

Before geometry, there’s a choice most teams underweight and then regret: the embedding model matters more than the database. Retrieval quality is set primarily by how good your embeddings are at capturing meaning in your domain. Choose and benchmark the embedding model on your own data first — long before you agonize over which vector store to buy. Get that wrong and no amount of downstream cleverness recovers it.

Then there’s chunking, the most underrated lever in the whole stack. You can’t embed a whole document as one point; you split it into pieces, and how you split determines what can ever be retrieved. The naive approach — cut every N tokens — is fast and dumb, slicing through the middle of a thought so that neither half retrieves well. Better approaches split at meaning boundaries, or respect the document’s actual structure: its headings, sections, tables, and lists. The chunk is the unit of retrieval, so a good chunk is a complete, self-contained idea, not a fragment. Garbage chunks produce garbage retrieval no matter how sophisticated everything after them is. Clean and de-duplicate before you embed; the boring work here pays back more than any flashy component later.

The retrieval stack: naive RAG is dead

The “hello world” of retrieval — embed the query, grab the nearest few chunks by cosine similarity, stuff them in, generate — is genuinely dead for production. It fails in two predictable ways that both trace back to lossy embeddings. It misses exact terms, because a part number or a specific name or a precise clause is exactly the detail compression discards. And it can’t reason across several facts at once. The modern pipeline exists to fix these failures, one layer at a time.

Query transformation comes first, and it’s optional but powerful: fix the user’s question before it hits the index. Real users ask vague, misspelled, under-specified questions. Expanding, rephrasing, or generating a hypothetical ideal answer to search against turns a bad query into a retrievable one. Don’t send raw human questions straight to your index.

Hybrid search is the core upgrade. Run semantic vector search (for meaning) and traditional keyword search (for exact terms) side by side, then merge the two ranked lists with a fusion step. This routinely beats either method alone by twenty to thirty-five percent, for the obvious reason: vectors catch “conceptually similar” and keywords catch “literally this exact string,” and real questions need both. A vector search can even treat a singular and a plural of the same word as meaningfully different; keyword search simply doesn’t care. Semantic search finds what you meant; keyword search guarantees you didn’t lose what you said.

Reranking is the layer most often called the secret sauce, and it makes retrieval a two-stage affair. First you cast a wide, cheap net — pull a few dozen candidate chunks. Then a reranker — a model that looks at the query and each candidate together, rather than comparing pre-computed points — scores true relevance far more precisely and keeps only the best handful for the context. It’s expensive per item, which is exactly why you run it only on the shortlist, not the whole corpus. The wide-then-narrow shape is the whole trick: retrieve broadly and imprecisely, then judge narrowly and precisely.

Put together, the pipeline is: query → transform → retrieve widely with hybrid search → fuse → rerank down to a few → generate. It adds only a few hundred milliseconds. And every layer earns its place by fixing a specific failure of the one before it — which is precisely what naive RAG left out.

Where the vectors live: the store as a production decision

The vector store gets outsized attention because it’s the shiny new kind of database, but keep it in proportion: it is one layer in a stack that also includes query handling, hybrid routing, reranking, and permission filtering. And in 2026 there is no universal winner — the right choice is a production-engineering trade-off, not a leaderboard.

The axes that actually decide it are retrieval quality, metadata filtering, tail latency under real concurrency, operational complexity, the security and multi-tenancy model, freshness requirements, and — frequently the deciding factor — whether your vectors ought to live right next to your existing data.

Which points to a pragmatic default worth stating plainly: if you already run Postgres, start with its vector extension. It’s mature, it’s used in production at serious scale, and it keeps your vectors alongside your relational data in one system your team already operates — no second database, no extra bill, no extra thing to secure. You may well never outgrow it. Reach for a purpose-built vector database only when scale, first-class hybrid search, or hard multi-tenant isolation genuinely forces the move. This is Chapter 4’s “add only what you need” and Chapter 3’s “don’t take on a system you’ll regret,” applied to retrieval — and it’s not a vendor preference, it’s an architecture heuristic: start with what you have, add complexity when a requirement demands it.

The mistake to avoid is choosing on a benchmark screenshot or a popular tutorial rather than on your real filters, query volume, security model, and operational reality — and then treating the decision as permanent. It isn’t; migrating a retrieval layer later is far more tractable than the marketing implies. And note the quiet bridge to the next section: where the store runs is also where your documents and their embeddings physically live, which for many enterprises matters more than any feature on the comparison chart.

The enterprise realities that break demos

A grounding demo works in an afternoon. A grounding system a regulated enterprise will actually run is a different animal, and four realities are what separate them.

Permissions are the one that gets companies in trouble. Access control has to be enforced at the moment of retrieval, not merely at the interface. If you filter what’s displayed but not what’s fetched, the model can read — and then paraphrase, summarize, or quote in its answer — documents the user was never cleared to see. The interface hid the file; the model leaked its contents. The discipline is permission-aware retrieval: every query is scoped, at retrieval time, to exactly what this user is allowed to see, so the model is structurally incapable of grounding on forbidden data. Get this wrong and no amount of polish saves you.

Freshness is a feature, not an afterthought. Your data changes, and a stale index confidently serves last quarter’s answer with a straight face. Re-indexing at scale is costly and error-prone — it has cost real companies real customers — so you need incremental updates and an honest sense of how current your index actually is. An answer that was true when you indexed and false today is still a wrong answer.

Residency and governance extend to the embeddings. Here is the subtlety teams miss: embeddings are derived data. They are not “just numbers” — they can leak information about the source text, and they fall under the same residency, retention, and governance rules as the documents they came from. Where your vectors live is a data-residency decision. Govern the embeddings exactly as you govern the source. Chapters 4 and 11 carry this further.

And there’s a failure the model will actively hide from you. Retrieve the wrong context, or nothing relevant, and the model does not stop — it answers anyway, confidently, grounding its hallucination in whatever you happened to hand it. Grounding without guardrails doesn’t remove hallucination; it makes hallucination sound more authoritative, now with citations attached. The defenses are concrete: make the model cite its sources, verify that those citations actually support the claim, and design the system so that empty or weak retrieval produces an honest “I don’t know” rather than a confident fabrication. Which is impossible to guarantee unless you can measure it — the subject we close on.

Agentic and graph retrieval, and how you know it works

Two advanced shapes go beyond the static pipeline — and, following Chapter 6’s discipline, you reach for them only when a real problem demands it.

Agentic RAG stops treating retrieval as a single up-front step. Instead, the model decides when to retrieve, what to search for, and whether to search again after seeing initial results — retrieval as a tool call, from Chapter 6. It can reformulate its own query, search several times, and reason across what it finds. That’s powerful for open-ended, multi-step questions, and it’s also more expensive and less predictable — so use it when a fixed pipeline provably can’t handle the variance, not by default.

GraphRAG answers the questions flat chunks can’t. Ask “which of our vendors share a dependency that’s currently flagged as at-risk,” and no amount of similarity search over isolated paragraphs will connect the dots, because the answer lives in relationships across documents. A knowledge graph — entities and the links between them — gives the retriever structure to traverse those relationships and reason across multiple hops. It’s more work to build and keep current, and it’s the right tool for connected, multi-hop domains.

And now the part most teams skip and then bitterly regret: evaluation. Grounding is genuinely hard to evaluate because it isn’t one signal, it’s four, and each needs its own method. Retrieval recall: did you fetch the right chunks at all? Context relevance: was what you fetched actually on point, or padded with noise? Answer faithfulness: did the answer stick to the retrieved context, or wander off and invent? Answer correctness: was the final answer, all things considered, actually right? A system can ace one and fail another — flawless retrieval feeding an unfaithful answer, or a faithful answer built dutifully on the wrong chunks. Measure all four or you are shipping something you cannot diagnose when it degrades, and it will degrade, silently, exactly like the context rot of Chapter 5. This is where Chapter 7 hands off to Chapter 9: grounding you can’t evaluate is grounding you can’t trust, so build the evaluation alongside the pipeline, not after it breaks in front of a user.

Step back and the shape of the work is clear, and a little humbling. Great grounding is unglamorous: good chunks, hybrid retrieval, a reranker, permission-scoping, freshness, honest citations, and relentless measurement. There is no single clever trick. But that unglamorous discipline is precisely the moat — because it is exactly the part the ninety-five percent skip on their way to a demo that never survived a real question.

The other lever: fine-tuning, and when to reach for it

Everything so far in this chapter adapts a model to your world by feeding it the right context at the moment you ask. There’s a second lever that adapts the model itself — fine-tuning — and because most teams reach for it too early and for the wrong reason, it’s worth being blunt about when it earns its place.

The single heuristic that settles most cases is this: fine-tuning is for form, not facts. It shapes behavior — a house tone, a strict output schema, a domain vocabulary, a tool-call format, a refusal pattern — the things you want the model to do consistently without being told every time. It is genuinely bad at injecting knowledge, and worse than useless for knowledge that changes, because a fact baked into the weights goes stale, can’t be cited, and can’t be permission-scoped. Knowledge is retrieval’s job — this entire chapter. So if your complaint is “it doesn’t know our facts,” you don’t have a fine-tuning problem; you have the retrieval problem you just spent a chapter learning to solve.

The right order of operations, and the field is unusually agreed on this, is prompt, then RAG, then fine-tune, then distill — and you earn each step by exhausting the one before it. Prompting and solid retrieval are hours of reversible work; if they get you ninety percent of the way, the final ten rarely repays a training pipeline’s ongoing cost. So the honest answer to “should we fine-tune?” is, most of the time, “not yet” — have you actually run out of road on prompting and retrieval, and do you have an evaluation (Chapter 9) that shows the base model has plateaued? Absent that, fine-tuning is a solution shopping for a problem.

When it does earn its place, the signal is a stable task with a known-good output the base model can’t hold reliably — a rigid schema, a brand voice, a narrow domain register — backed by that eval and a few hundred to a few thousand genuinely clean examples (curated beats voluminous, every time). One more case deserves naming: compression. A smaller open-weight model, fine-tuned on your narrow task, can match a frontier model on that task at a fraction of the inference cost and latency — the cost lever of Chapters 5 and 10. And the two levers compose rather than compete: a common sweet spot is a fine-tuned small model for form with retrieval on top for facts.

On method, at the level this book cares about: you rarely retrain every weight — full fine-tuning is expensive and usually overkill. The accessible path is parameter-efficient, freezing the base model and training a small adapter on top (the family known as LoRA, and its quantized cousin that fits on a single GPU), cheap enough to iterate on and small enough to serve beside the base and swap in and out. The labs on leapfrog.lerias.org walk the mechanics; the book’s job is the decision.

And the part nobody quotes you on: the operational tax. Fine-tuning is not a one-time task, it’s a maintained pipeline. Adapters need versioning and rollback and a retraining cadence; your training data and configs are code (Chapter 8); and — the real sting — when your provider updates the base model underneath you, your adapter can silently degrade, which is the prompt drift of Chapter 8 wearing a new coat. So you plan periodic revalidation against your eval set, and you budget several times the training cost for the year of ownership that follows. The real cost was never the GPU hours. It’s the evaluation, the data curation, and the lifecycle. Reach for fine-tuning deliberately and eyes-open — form not facts, earned not reflexive — and most of the time you’ll find the prompting and retrieval you already built were enough.