Applied GenAI Curriculum for AI PMs  ·  Depth Layer  ·  Architecture Thread

Unit 17 — RAG Internals: Retrieval Quality

Term-based vs. embedding retrieval, chunking, reranking, query rewriting, and the trade-offs — enough to follow a retrieval-quality debate and know which lever the team is pulling, at what cost.

Tags
architecturedata-cost
Layer
DEPTH — when a RAG system underperforms and the team debates chunking / embeddings / rerankers
Objective
Follow a retrieval-quality debate and understand which lever the team proposes to pull — and at what cost.
Depends on
Unit 05 (RAG as retriever + generator). Pull in when a RAG system underperforms.

Why open up the retriever

Unit 05 said a RAG system is a retriever plus a generator, and that its quality rides on the retriever. This unit opens the retriever box — because when a RAG feature gives wrong or thin answers, the fix is almost always in retrieval, and the team will debate chunking, embeddings, and rerankers. You won't tune these, but you should follow the argument and know what each lever buys and costs.

The reframe: a wrong RAG answer is usually a retrieval failure If the right document never reaches the model, no prompt or model swap saves the answer. So "our RAG is bad" should trigger "show me the retrieval quality" before anything else. This unit is the vocabulary for that conversation.

1. The two ways to find documents

All retrieval ranks documents by relevance to a query. Two families do the ranking differently:

Lexical / keyword

Term-based retrieval

Matches on words. Ranks by how often a query's important terms appear (term frequency), weighting rarer terms higher (inverse document frequency) — the TF-IDF idea, industrialized as BM25 / Elasticsearch.

Strength: fast, cheap, strong out of the box. Weakness: literal — misses meaning, and hard to improve.

Semantic / meaning

Embedding-based retrieval

Converts chunks and the query into embeddings (vectors capturing meaning), stores them in a vector database, and fetches the k nearest. Ranks by meaning, not words.

Strength: handles natural queries, improvable via finetuning. Weakness: slower, costlier, can obscure exact keywords (error codes, product names).

Term-basedEmbedding-based
SpeedMuch faster (index & query)Query embedding + vector search can be slow
QualityStrong out of the box, hard to improve; can grab wrong sense of a word (e.g. "transformer" the device vs. the model)Can surpass term-based with finetuning; understands intent
CostCheapEmbedding generation, vector storage & search can be expensive — sometimes a fifth to half of model-API spend
Why production usually runs both: hybrid search Because the two fail in opposite ways, most real systems combine them (hybrid search). Common pattern: a cheap term-based pass fetches candidates, then an expensive semantic pass reranks to find the truly relevant ones — e.g. grab every doc containing "transformer," then use vector search to keep only the neural-network ones. (Parallel ensembles merged by rank fusion are the other pattern.) When a team says "we added hybrid search," this is what they mean.

The deeper machinery — approximate-nearest-neighbor algorithms (HNSW, IVF, LSH…) that make vector search fast — is real but the engineers' concern. The trade you should know: a richer index gives more accurate, faster queries but costs more time and memory to build (which bites when data changes often).

2. The optimization levers (and what each costs)

When retrieval underperforms, these are the four dials the team reaches for. Each improves relevance but spends something.

Chunking strategy

How documents are split for indexing — fixed size (characters/words/sentences/paragraphs), recursive splitting, or format-aware (code, Q&A pairs). Overlap between chunks avoids cutting key context mid-thought ("I left my wife / a note").

The trade: smaller chunks fit more diverse info in context but risk losing information and double the embeddings to store/search; larger chunks preserve context but retrieve less precisely. No universal best — it's tuned by experiment.

Reranking

Re-order the retriever's candidates with a more precise (pricier) scorer, or by recency for time-sensitive apps (news, email, markets). Especially useful to shrink the set to fit the model's context.

The trade: better top results for extra compute per query. Order matters less than in search — as long as a doc is included, its exact rank is secondary (though beginning/end of context are attended best).

Query rewriting

Rewrite an ambiguous query to stand on its own before retrieval. "How about Emily Doe?" following a question about John Doe must become "When did Emily Doe last buy from us?" — often done with another model.

The trade: big relevance win on multi-turn chat, but adds a model call and can get hairy with identity resolution ("how about his wife?" needs a lookup — and must admit when it can't resolve rather than hallucinate a name).

Contextual retrieval

Augment each chunk to make it findable: metadata (tags, keywords, extracted entities like an error code), the questions it answers ("how to reset password?" → "I can't log in"), or a short generated blurb situating the chunk within its source document.

The trade: notably better recall, especially for chunks that lost context when split — at the cost of extra indexing work (and model calls if you generate the context).

3. How to tell if retrieval is actually the problem

The whole debate is unwinnable without measurement. Retrieval has its own metrics, separate from the final answer:

Context precisionOf the documents retrieved, what fraction are relevant? (Cheap to compute — just judge the retrieved set against the query; an AI judge can do it.)
Context recallOf all relevant documents, what fraction did you retrieve? (Harder — needs every doc in the corpus labeled for relevance to the query.)
Ranking metricsNDCG, MAP, MRR — if you care whether the most relevant docs rank first.
Evaluate the RAG system in three places Retrieval quality (the metrics above), the embeddings themselves (do more-similar docs sit closer?), and the end-to-end answer (does the retrieved context actually produce a good response?). A retriever is only "good" if it helps the whole system answer well — so component metrics and end-to-end evals both matter (Units 09, 15).

4. RAG beyond text

External knowledge isn't only documents. Two extensions come up:

Multimodal RAGRetrieve images/video/audio alongside text — by their metadata (captions, titles) or, for content-based matching, via a multimodal embedding model that puts text and images in the same vector space.
Tabular RAG (text-to-SQL)Many questions need a database, not a document. The flow differs entirely: translate the natural-language question into a SQL query, run it, and feed the result back. "How many Fruity Fedoras sold last week?" becomes a SELECT SUM…, not a similarity search.

Worth recognizing because "add RAG" can quietly mean "build a text-to-SQL agent," which is a very different project with different failure modes.

What "good" looks like after this unit

You can now:

That covers retrieval depth. The other architecture-thread deep dives are agents (Unit 18) and finetuning (Unit 19), pulled in when those projects are on the table.