Applied GenAI Curriculum for AI PMs · Depth Layer · Architecture Thread
Term-based vs. embedding retrieval, chunking, reranking, query rewriting, and the trade-offs — enough to follow a retrieval-quality debate and know which lever the team is pulling, at what cost.
Unit 05 said a RAG system is a retriever plus a generator, and that its quality rides on the retriever. This unit opens the retriever box — because when a RAG feature gives wrong or thin answers, the fix is almost always in retrieval, and the team will debate chunking, embeddings, and rerankers. You won't tune these, but you should follow the argument and know what each lever buys and costs.
All retrieval ranks documents by relevance to a query. Two families do the ranking differently:
Matches on words. Ranks by how often a query's important terms appear (term frequency), weighting rarer terms higher (inverse document frequency) — the TF-IDF idea, industrialized as BM25 / Elasticsearch.
Strength: fast, cheap, strong out of the box. Weakness: literal — misses meaning, and hard to improve.
Converts chunks and the query into embeddings (vectors capturing meaning), stores them in a vector database, and fetches the k nearest. Ranks by meaning, not words.
Strength: handles natural queries, improvable via finetuning. Weakness: slower, costlier, can obscure exact keywords (error codes, product names).
| Term-based | Embedding-based | |
|---|---|---|
| Speed | Much faster (index & query) | Query embedding + vector search can be slow |
| Quality | Strong out of the box, hard to improve; can grab wrong sense of a word (e.g. "transformer" the device vs. the model) | Can surpass term-based with finetuning; understands intent |
| Cost | Cheap | Embedding generation, vector storage & search can be expensive — sometimes a fifth to half of model-API spend |
The deeper machinery — approximate-nearest-neighbor algorithms (HNSW, IVF, LSH…) that make vector search fast — is real but the engineers' concern. The trade you should know: a richer index gives more accurate, faster queries but costs more time and memory to build (which bites when data changes often).
When retrieval underperforms, these are the four dials the team reaches for. Each improves relevance but spends something.
How documents are split for indexing — fixed size (characters/words/sentences/paragraphs), recursive splitting, or format-aware (code, Q&A pairs). Overlap between chunks avoids cutting key context mid-thought ("I left my wife / a note").
The trade: smaller chunks fit more diverse info in context but risk losing information and double the embeddings to store/search; larger chunks preserve context but retrieve less precisely. No universal best — it's tuned by experiment.
Re-order the retriever's candidates with a more precise (pricier) scorer, or by recency for time-sensitive apps (news, email, markets). Especially useful to shrink the set to fit the model's context.
The trade: better top results for extra compute per query. Order matters less than in search — as long as a doc is included, its exact rank is secondary (though beginning/end of context are attended best).
Rewrite an ambiguous query to stand on its own before retrieval. "How about Emily Doe?" following a question about John Doe must become "When did Emily Doe last buy from us?" — often done with another model.
The trade: big relevance win on multi-turn chat, but adds a model call and can get hairy with identity resolution ("how about his wife?" needs a lookup — and must admit when it can't resolve rather than hallucinate a name).
Augment each chunk to make it findable: metadata (tags, keywords, extracted entities like an error code), the questions it answers ("how to reset password?" → "I can't log in"), or a short generated blurb situating the chunk within its source document.
The trade: notably better recall, especially for chunks that lost context when split — at the cost of extra indexing work (and model calls if you generate the context).
The whole debate is unwinnable without measurement. Retrieval has its own metrics, separate from the final answer:
| Context precision | Of the documents retrieved, what fraction are relevant? (Cheap to compute — just judge the retrieved set against the query; an AI judge can do it.) |
| Context recall | Of all relevant documents, what fraction did you retrieve? (Harder — needs every doc in the corpus labeled for relevance to the query.) |
| Ranking metrics | NDCG, MAP, MRR — if you care whether the most relevant docs rank first. |
External knowledge isn't only documents. Two extensions come up:
| Multimodal RAG | Retrieve images/video/audio alongside text — by their metadata (captions, titles) or, for content-based matching, via a multimodal embedding model that puts text and images in the same vector space. |
| Tabular RAG (text-to-SQL) | Many questions need a database, not a document. The flow differs entirely: translate the natural-language question into a SQL query, run it, and feed the result back. "How many Fruity Fedoras sold last week?" becomes a SELECT SUM…, not a similarity search. |
Worth recognizing because "add RAG" can quietly mean "build a text-to-SQL agent," which is a very different project with different failure modes.
That covers retrieval depth. The other architecture-thread deep dives are agents (Unit 18) and finetuning (Unit 19), pulled in when those projects are on the table.