RAG Retrieval Reliability: Why I Stopped Trusting It

The Bug That Made Me Distrust RAG

I built a private health AI for a family member. It is a multi-agent medical team I built, with specialist agents that each answer questions about a specific domain: vitals, medications, history, lab trends. You ask a question, the right specialist responds, and a synthesis layer pulls it together.

It worked well. Until it didn't, in a way that taught me to stop trusting rag retrieval reliability for the facts that actually matter.

Here was the failure. Ask the system a general question, something like "how am I doing overall?" and an agent would respond, with total confidence, "I don't see a BMI in your chart." The BMI was right there in the data. Current, classified, sitting in the same record the agent had access to.

This is not a missing-data problem. The data was present. The retrieval layer simply failed to surface it, so the model never saw it, and then it told the user the fact didn't exist.

Sit with that for a second. The AI was not wrong about the answer. It was wrong about whether it had the answer. Those are two completely different failures, and the second one is far more dangerous, because it looks like honesty. "I don't see X" sounds careful. It sounds trustworthy. It is the model telling you it checked.

In a health context, a false "I don't see X" isn't a typo. It is a wrong decision waiting to happen. A BMI in the obesity range that gets reported as missing. A medication that doesn't show up when it should. The whole point of putting AI on top of your data is that it uses all of it, especially the parts that matter most.

So here is the buyer doubt I want to address directly: can you actually trust AI to use every relevant fact in your data? Or will it confidently miss the ones that matter and tell you they were never there?

Why Vector Search Misses the Facts That Matter

To fix this I had to understand why it happens. It is not a bug in any normal sense. It is how retrieval is designed to work.

How vector search misses critical facts on general queries

Similarity is not relevance

RAG retrieves chunks of your data by semantic similarity to the query. It embeds your question, embeds your documents, and returns the top-k chunks whose vectors sit closest to the question's vector.

That works beautifully when the query and the fact use similar language. Ask "what medications am I on?" and chunks about medications light up. Great.

The problem is that similarity is not the same as relevance. A fact can be enormously relevant to a question while sharing almost no semantic overlap with the words used to ask it. Vector search has no way to know that. It optimizes for lexical and semantic closeness, not for importance.

General questions are the worst case

Now ask a broad question: "how am I doing?" There is nothing in that phrase that resembles a specific BMI reading or a drug allergy. The single most important fact in the record, the one a clinician would surface first, doesn't lexically or semantically match the query at all.

So vector search returns the top-k closest chunks, and the critical vital just isn't among them. The model only ever sees what was retrieved. If the BMI wasn't in the top-k, the model genuinely doesn't have it, and it honestly reports it as missing.

This is the core mechanism behind vector search missing facts. Similarity search is probabilistic by design. It optimizes for average relevance across a corpus, not for the guaranteed presence of any one specific high-stakes fact.

And here is the pattern that makes it dangerous: the more general the question, the less reliable retrieval becomes for narrow critical facts. The exact questions where you most want the AI to surface the important stuff are the questions where retrieval is least likely to. That is when RAG fails, and it fails quietly.

Some Facts Are Too Important to Leave to Retrieval

Once I understood the mechanism, the fix became obvious. There is a category of facts where "usually retrieved" is not good enough.

Foreground (inject) vs background (retrieve) fact classification across domains

In a health context, that category is small and well defined: latest BMI with its classification, current weight and height, blood pressure, heart rate, active medications, known allergies. These are foreground facts. They are relevant to almost any question about a person's health, and the cost of the model not seeing one is severe.

You do not want a probability distribution deciding whether the model sees a drug allergy. You want it there, every time, no exceptions.

The mistake most systems make is treating all context as equally retrievable. They throw everything into a vector store and let similarity sort it out. That is fine for the long tail of background information. It is reckless for the handful of facts that should never be absent.

This isn't unique to health. Every domain has its own version of these foreground facts:

• An account balance in a financial AI
• A contract expiration date in a legal AI
• Order status in a customer service AI
• Allergy information anywhere it exists

These are facts where being wrong about whether you have them causes a wrong decision. They belong in the foreground, always present, never dependent on whether an embedding happened to match the phrasing of a question.

This maps cleanly onto a principle I keep coming back to: let the model judge and let the code compute. The AI is great at reasoning over context. It is terrible as the mechanism that decides whether critical context is present at all. That decision should be deterministic. Code should guarantee the fact is in front of the model. The model should reason about it from there.

The Fix: Inject the Critical Facts Deterministically

So I stopped trying to make retrieval reliable for these facts and removed retrieval from the equation entirely.

A key-vitals block on every agent's context

I built a deterministic summary builder, a function I call getKeyVitalsSummary. It does one job: it constructs a structured block of the critical facts and prepends it to every agent's context, on every question, regardless of what vector search returned.

The block contains the latest BMI plus its obesity classification, current weight and height, blood pressure, heart rate, and short-term trends. Every specialist agent sees it, every time. There is no retrieval roll of the dice for the facts that matter most.

The "I don't see a BMI" failure became structurally impossible. The BMI is always there, sitting at the top of the context, before the model reasons about anything.

Why this beats tuning the retriever

The obvious alternative was to tune the retriever. Improve the chunking, adjust the embeddings, bump up k, add reranking. I could have spent weeks on it.

Deterministic injection vs tuning the retriever tradeoff

But you can chase recall forever and still hit edge cases. There is always some phrasing of some question where the critical fact falls just outside the top-k. With retrieval, you are managing a failure rate. With high-stakes facts, the acceptable failure rate is zero, and you cannot get to zero with a probabilistic system.

Deterministic injection is cheap, predictable, and auditable. I can look at the code and tell you exactly which facts are guaranteed present. I cannot do that with a retriever. This is the heart of the rag vs deterministic context decision: for a small set of always-critical facts, deterministic wins on every axis that matters.

The honest tradeoff is token cost. The block goes into every agent's context on every call, so I pay for those tokens whether or not the question needs them. It is a few hundred tokens. For facts where a single miss could cause a wrong health decision, that is not a cost, it is insurance. I wrote a deeper breakdown of this pattern in always inject the critical facts if you want the full reasoning.

Make the Model Verify Before It Says 'I Don't See It'

Injection solves the retrieval problem. It does not solve the model problem. Even with the BMI sitting in context, a model can still hallucinate a "missing data" claim. So I added two more layers.

The RECORD_CHECK_MANDATE

Every system prompt in the system includes a record-check mandate. The rule is simple: the model is forbidden from saying "I don't see X" without first confirming X is actually absent from the raw data it was given.

This is a guardrail, the same philosophy I use when constraining AI so it can't embarrass you. You do not hope the model behaves. You write the constraint into the prompt and make the unwanted behavior an explicit violation. "Before claiming any fact is missing, verify it is not present in the data above" is a small instruction that closes a large gap.

A synthesis stage that catches false negatives

The second layer is structural. The multi-agent design has a chief-synthesis agent that reviews the specialist outputs before anything reaches the user. Part of its job is to flag false missing-data claims across the specialists.

The layered defense architecture: deterministic injection + verification mandate + synthesis review

If one specialist says "no BMI on file" while the data clearly contains one, the synthesis agent catches the contradiction and corrects it before the user ever sees the false claim. One specialist's blind spot gets caught by a second independent review.

So there are two independent checks: the prompt-level mandate inside each agent, and the cross-agent review on top. A false negative has to survive both to reach the user.

This is not free. The mandate adds tokens, and the synthesis stage adds latency and cost. For a casual chatbot that would be over-engineering. For high-stakes facts where being wrong about what you have is unacceptable, it is the correct trade, and I make it deliberately.

When RAG Is Still the Right Tool

I want to be clear: I am not anti-RAG. I use it constantly, including in this same system.

For the long tail, retrieval is exactly right. Historical clinical notes, prior conversations, large document corpora, anything where you genuinely cannot know in advance what will be relevant to a given question. You cannot inject all of that into every context. It would not fit, and most of it would be noise. That is precisely the problem RAG was built to solve, and it solves it well.

The rule I follow is simple. Deterministically inject the small set of always-critical facts. Use RAG for the large body of situationally-relevant context. The two are complementary, not competing. Injection handles the foreground, retrieval handles the background, and together they cover the whole field.

Here is the test I give clients to decide which bucket a fact belongs in:

If a fact being missed would cause a wrong decision, inject it. If it is nice-to-have background, retrieve it.

A drug allergy being missed causes a wrong decision. Inject it. A note from a routine checkup eighteen months ago is useful context if it is relevant, harmless if it is not. Retrieve it.

That single question sorts almost everything cleanly. It also applies far beyond health. In a finance AI, the account balance gets injected and the transaction history gets retrieved. In a legal AI, the expiration date gets injected and the supporting clauses get retrieved. Same principle, different domain.

What This Means for Trusting AI With Your Data

So back to the doubt I opened with. Can you trust AI to use all your data?

Yes. But only if someone designed the system to guarantee it, not hope for it.

Out of the box, RAG will confidently miss things. It will tell you a fact isn't there when it is, and it will do it in a tone that sounds careful and trustworthy. Most vendors will not tell you this, partly because their demo runs on clean questions where retrieval happens to work, and partly because the failure is invisible until it costs you.

The difference between a demo and a production system is exactly this kind of plumbing. Knowing which facts cannot be left to probability. Building the deterministic backstop. Adding the verification mandate. Putting a synthesis stage on top to catch what slips through. None of that shows up in a slick demo. All of it shows up the first time the convenient default quietly fails.

That is the work I do as a Chief AI Officer. I find the places where the default behavior breaks in ways nobody noticed, and I engineer around them before they cost you a wrong decision. The rag retrieval reliability problem is one example. There are dozens like it hiding in any serious deployment.

If you are putting AI against data where being wrong about what you have is not an option, that is the conversation worth having.

Want to explore what AI could do for your business?

Book a free 30-minute strategy call. No pitch deck, no sales team, just a real conversation about your operations and where AI actually fits.

Book a Discovery Call