Retrieval and RAG for Developers: Grounding AI Answers in Your Own Data

Why RAG Answers Go Wrong

The pipeline is short: embed the question, search an index, put the top results in the prompt, ask the model to answer from them. Four steps, and the failure you notice — a wrong answer — can come from any of them. Before changing anything, find out which one.

Take the question that produced the bad answer and inspect the chunks that were actually retrieved. That single act separates most failures into one of four buckets:

• The answer isn't in the corpus. Nothing was indexed that contains it. No amount of retrieval tuning fixes this, and the correct behaviour is for the model to say it doesn't know.
• The answer is in the corpus but wasn't retrieved. A recall failure — often caused by chunking, embedding, query wording, or index settings. This is a common bucket and a fixable one.
• The answer was retrieved but ranked too low. It came back at position 14 while you passed the top 5. A ranking problem, not a recall problem, and it has different fixes.
• The answer was retrieved, ranked well, and the model still got it wrong. Now — and only now — you have a generation problem worth prompt engineering.

Teams skip this diagnosis because it requires logging retrieval results, and it is faster to edit a prompt than to build a debugging surface. It is also why RAG systems plateau: every fix targets the last step of a four-step pipeline.

Chunking Is a Design Decision

Chunking is where much of retrieval quality is won or lost, and it gets too little attention because a common tutorial default — split every 1,000 characters, overlap by 200 — runs without error. It runs without error while cutting tables in half and separating a heading from the paragraph that explains it.

A chunk is the unit your system can retrieve. It should be small enough to be specific and large enough to be self-contained, which means it should follow the document's structure rather than a character count:

• Split on structure first. Headings, sections, list boundaries, table boundaries. A Markdown or HTML document already tells you where the seams are — use them instead of guessing.
• Fall back to size limits within a section. If a section exceeds your target size, split it at paragraph boundaries, not mid-sentence.
• Keep tables and code blocks intact. Half a table retrieves as noise. If a table is too large, repeat its header row in each piece.
• Prepend inherited context. A chunk that reads "This limit does not apply to enterprise accounts" is useless alone. Prefix it with the document title and heading path so it can stand by itself.

That last point matters more than overlap tuning. Overlap is a blunt way of preserving context at the seams; carrying the heading path into the chunk text preserves it deliberately.

{
  "chunk_id": "handbook-2026#leave/parental/eligibility-3",
  "text": "Employee Handbook 2026 > Leave Policy > Parental Leave > Eligibility\n\nEmployees become eligible after 90 days of continuous employment. This requirement does not apply to transfers from an acquired entity.",
  "metadata": {
    "source_title": "Employee Handbook 2026",
    "heading_path": ["Leave Policy", "Parental Leave", "Eligibility"],
    "source_url": "/handbook/leave#parental",
    "updated_at": "2026-06-30",
    "visibility": "all-employees"
  }
}

The metadata is not decoration. source_url is how the answer cites itself, updated_at is how you detect stale content, and visibility is how you keep the wrong person from retrieving it — all covered below.

Embeddings and the Index

An embedding model turns text into a vector — a list of numbers positioned so that texts with similar meaning land near each other. Retrieval is then a nearest-neighbour search: embed the query, find the closest chunk vectors, return them. That's the whole idea, and two practical consequences follow from it.

First, vectors from different embedding spaces are not comparable. Each model defines its own vector space and may use a different dimension count. Switching to a model or version that produces a different space means re-embedding your entire corpus. Plan it as a reindex-and-switch migration rather than a config change, and keep the old index serving traffic until the new one is verified. The same expand-backfill-switch-contract shape described in The AI Database Migration Plan applies here.

Second, embeddings match meaning rather than exact strings. They are useful for paraphrase but can be weaker on identifiers: error codes, SKUs, version numbers, function names, and surnames are queries where a plain keyword index may win. This is why hybrid retrieval is worth testing against a purely vector-based baseline.

Where to Put the Vectors

You do not need a specialised database to start. If your data already lives in Postgres, the pgvector extension stores vectors as a column type and supports approximate-nearest-neighbour indexes (HNSW and IVFFlat), which means retrieval can join directly against the rows that carry your permissions and timestamps. That single property — filtering and searching in one query, in one transactional store — solves more real problems than raw search speed does at small and medium corpus sizes.

A dedicated vector database earns its place when the corpus outgrows what your primary database should be doing, when you need multi-tenant index isolation, or when you want managed reranking and hybrid search rather than assembling them yourself. The trade you accept is a second system to keep consistent with the source of truth: every document update, deletion, and permission change now has to reach two places.

Retrieval Quality You Can Measure

"The answers feel better now" is not a result you can build on. Retrieval is measurable. A set of thirty examples is enough to expose obvious failures and start comparing changes, though it is not enough to claim reliable performance across every query type.

Build a gold set: real questions from real users, each paired with the chunk or document that genuinely contains the answer. Real questions matter — the ones you invent are phrased in the vocabulary of your documentation, which is exactly the vocabulary your users don't have. Then measure the retrieval step alone, with no model in the loop:

Include questions whose answers are deliberately not in your corpus. Without them you optimise for confident coverage and never notice that your system cannot say "I don't know" — the single most damaging failure mode in a documentation assistant. Wire these measurements into the same harness described in AI Evals in Production so a chunking change or an embedding upgrade shows its effect before it ships.

Two Fixes to Test Before Replacing the Embedding Model

Hybrid search. Run a keyword search and a vector search, then merge the two ranked lists. Reciprocal rank fusion is the standard way to do it and needs no score normalisation — each result is scored by its position in each list, so a chunk ranked well by either method surfaces:

function fuse(keywordHits, vectorHits, k = 60) {
  const scores = new Map();

  for (const list of [keywordHits, vectorHits]) {
    list.forEach((hit, index) => {
      const prev = scores.get(hit.chunkId) ?? 0;
      scores.set(hit.chunkId, prev + 1 / (k + index + 1));
    });
  }

  return [...scores.entries()]
    .sort((a, b) => b[1] - a[1])
    .map(([chunkId, score]) => ({ chunkId, score }));
}

Reranking. Retrieve a larger candidate set — for example, 50 chunks — then score each candidate against the query with a reranking model that reads the pair together rather than comparing two independent vectors. It is slower per candidate, which is why it runs on a bounded candidate set instead of the whole corpus, and it directly targets the "found it but ranked it fourteenth" failure.

Grounding the Answer

Retrieval gives the model the right text. It does not stop the model from blending that text with what it already believes, and a plausible sentence assembled from training data is indistinguishable from a sourced one unless you make the source visible.

Three rules do most of the work:

1. Label every chunk in the prompt with its source. Give each retrieved passage an explicit identifier and require the answer to reference the ones it used. An answer that cites nothing was not grounded in anything.
2. Make "not in the provided context" a first-class output. State it as an instruction, show it in the response format, and test for it. Models default to helpfulness; refusal has to be an option you built.
3. Surface conflicts instead of averaging them. When two retrieved passages disagree — an old policy and its replacement — the useful answer names the disagreement and prefers the newer source. Silently picking one is the worst outcome, because it looks like certainty.

Answer the question using only the numbered sources below.

Rules:
- Every factual claim must cite the source it came from, as [1], [2].
- If the sources do not contain the answer, reply exactly:
  "The provided sources don't cover this." Then say what
  information would be needed. Do not use outside knowledge.
- If two sources conflict, say so, cite both, and prefer the
  one with the more recent updated_at date.
- Quote exact figures, dates, and identifiers rather than
  paraphrasing them.

Sources:
[1] (updated 2026-06-30) Employee Handbook 2026 > Leave Policy >
    Parental Leave > Eligibility
    Employees become eligible after 90 days of continuous employment...

[2] (updated 2025-11-02) Employee Handbook 2025 > Leave Policy...

Question: {user_question}

Then verify the citations mechanically. If the model cites [3] when you supplied two sources, or cites a chunk whose text does not contain the figure it quoted, that is a detectable defect — not something to leave to a human noticing it later.

Permissions, Freshness, and Cost

A retrieval index is a copy of your data with its own access path. Everything your application enforces on the original has to be enforced again here, and missing that second enforcement point is a common way for internal assistants to expose the wrong content.

Filter in the Query, Never in the Prompt

Access control belongs in the retrieval query as an authorization predicate, so unauthorized chunks cannot be returned to the application or placed in the prompt — the same principle as provider-enforced storage permissions in Adding File Uploads to Your Vibe Coded App. Two patterns to avoid:

• Instructing the model to ignore documents the user may not see. The confidential text is already in the prompt. You have disclosed it and are hoping the output hides it.
• Retrieving broadly and filtering the results afterwards. Better, but it silently degrades quality: if seven of your ten hits get dropped, the user gets three, and your recall measurements no longer describe what they experience. Filter first, then rank.

Permission changes also have to propagate. When someone leaves a team or a document's visibility narrows, the index must be updated, because a stale visibility field is an access-control bug that no code review will catch. The same applies to deletion: a document removed from the source system but left in the index is still answerable.

Keep security semantics separate from search-engine execution details. In pgvector, filtering with an approximate index is applied after the index scan, which can return too few authorized results unless you tune the candidate scan or enable iterative scans. The authorization predicate must still be in the database query; then benchmark recall on the filtered queries your users actually run. Never compensate for a short result set by issuing an unfiltered query.

Freshness

Prefer reindexing on change, with a durable event or job created alongside the source update. A nightly full rebuild is simple and can be fine for a small corpus, but it also permits up to a day of staleness. Carry updated_at into the prompt so the model can prefer recent sources, and consider excluding content past a staleness threshold rather than serving it silently.

Cost

Retrieval changes the shape of your token spend. Corpus embeddings are paid or computed when documents are indexed and again when they change or the embedding space changes. Query-embedding cost varies with query length but is usually small beside generation; the larger recurring cost is often the retrieved context sent with every answer. Two consequences worth planning for:

• Larger k is not free. If chunk sizes are similar, going from 5 chunks to 20 roughly quadruples retrieved-context tokens per request for a recall gain a reranker may deliver more cheaply. Measure both.
• Stable context caches well. Instructions and system context that don't change per query are good candidates for prompt caching; retrieved chunks that differ every request are not. AI Cost Modeling covers the arithmetic.

And before any of it: if the corpus never grows and the answers come from a handful of stable documents, consider whether you need retrieval at all. Putting those documents in the prompt is cheaper to build, cheaper to run, and has no index to keep in sync. Should This Feature Use AI? makes the broader version of this argument.

Copy-Paste Prompt: Audit a Retrieval Pipeline

Use this on a RAG implementation that answers incorrectly and you don't yet know why:

I have a retrieval pipeline in [files/functions] that answers questions
over [describe the corpus: size, format, update frequency]. It returns
wrong or invented answers for questions like [2-3 real examples].

Before changing any prompt, help me isolate where it breaks:

1. Add retrieval logging: for each request, record the query, the
   retrieved chunk IDs, their scores, and their rank order. Log
   identifiers and metadata, not raw document text.
2. For my example questions, tell me whether the correct source is
   (a) absent from the index, (b) present but not retrieved,
   (c) retrieved but ranked below the cutoff, or (d) retrieved and
   ranked well. Show the evidence for your conclusion.
3. Review the chunking: does it split on document structure, keep
   tables and code blocks intact, and carry title and heading path
   into the chunk text? Show me a real chunk from my corpus as proof.
4. Check that access control is applied as a metadata filter in the
   retrieval query, not as a prompt instruction and not as a
   post-filter on the results.
5. Propose a gold set of 30 questions from my actual query logs,
   including at least 5 whose answers are NOT in the corpus, and a
   script that reports recall@5 and recall@50 against it.

Report findings and the smallest fix for each, in priority order. Do
not rewrite the answer prompt until we know retrieval is the problem.
State which conclusions you verified and which are assumptions.

Related Guides