Applied GenAI Curriculum for AI PMs · Core Sequence · Eval Thread 2/5

Unit 09 — Designing the Eval Pipeline

The three-step pipeline design — evaluate every component, write an evaluation guideline, then define methods and data — turning "we need evals" into a concrete, buildable plan.

From "we need evals" to a plan

Unit 06 argued evals are the whole game; Unit 07 defined what "good" means. This unit is the bridge to doing it: a three-step recipe for designing an evaluation pipeline you can actually build. It targets open-ended tasks (close-ended ones are easier and fall out as a simplification). By the end you should be able to sketch a component-level eval plan for a real system — which is precisely the capstone deliverable in Unit 13.

Evaluate all components in the system

Decompose the system and measure each piece independently, per turn and per task — not just the final output.

Create an evaluation guideline

Define what "good" (and "bad," and "out of scope") means, with criteria, scoring rubrics, and a mapping to business impact. The most important step.

Define evaluation methods and data

Pick a method per criterion, then curate and slice annotated evaluation sets sized for reliable results.

Step 1 — Evaluate every component, not just the output

Real systems are multi-component and multi-turn. If you only measure the end result, a failure tells you that something broke, not where. Take a system that pulls a person's current employer from a résumé PDF in two steps: (1) extract text from the PDF, (2) extract the employer from that text. A wrong answer could come from either. Measure them separately — text-extraction by similarity to ground truth, employer-extraction by accuracy given correct text — and you know exactly which to fix.

Evaluate at multiple granularities, and mind the crucial distinction:

Turn-based	Quality of each individual output/response. A "turn" can span multiple steps.
Task-based	Did the system actually accomplish the user's goal — and in how many turns? Solving a problem in 2 turns vs. 20 is a huge difference.

Task-based is what users actually care about Users don't care if any single response was elegant; they care whether the system got their job done. Task-based evaluation is more important — but harder, because task boundaries are fuzzy (is the next message a follow-up or a new task?). Push your team to measure task completion, not just per-response quality.

Step 2 — The evaluation guideline (the most important step)

An ambiguous guideline yields ambiguous scores. If you can't say what a bad response looks like, you can't catch one. Three sub-tasks:

Define what good means — including what it shouldn't do

The hardest part usually isn't judging an output; it's defining "good." A correct response isn't always a good one: to a job-fit question, "You are a terrible fit" may be accurate but useless — a good response explains the gap and how to close it. Also define scope: should a support bot answer questions about an upcoming election? If not, specify how to detect out-of-scope inputs and how to respond. In practice, teams average around two to three criteria per application — e.g. for support: relevance, factual consistency, safety.

Create scoring rubrics with examples

For each criterion pick a scoring scheme (binary 0/1; 1–5; −1/0/1 for contradiction/neutral/entailment) and write a rubric with concrete examples: what earns a 1, and why? Then validate the rubric with humans — if people can't apply it consistently, it's too ambiguous and needs refining. This back-and-forth is the backbone of a reliable pipeline (and the rubric later doubles as annotation guidance for finetuning data).

Tie evaluation metrics to business metrics

A metric only matters in the context of the business goal. "80% factual consistency" is meaningless until you translate it into impact:

80% factual

30%

of support automatable

90% factual

50%

automatable

98% factual

90%

automatable

This mapping is what makes investment decisions defensible — you can see what a metric gain is worth. Set a usefulness threshold too (the minimum score to be useful at all), and know the underlying business metric you're serving (stickiness, engagement) — while staying aware that optimizing engagement can push a product toward addictive or extreme content.

Step 3 — Methods and data

Select a method per criterion — and mix them

Different criteria want different methods: a small specialized classifier for toxicity, semantic similarity for relevance, an AI judge for factual consistency against context. You can mix and match on the same criterion to balance cost and confidence — e.g. a cheap classifier on 100% of data plus an expensive AI judge on a 1% sample. Use logprobs when available to read model confidence (three classes all at 30–40% = unsure; one at 95% = confident). And don't be afraid to keep human evaluation in the loop, even in production — many teams treat human judgment as the North Star, hand-reviewing a daily sample to catch drift.

Experimentation vs. production evals differ In experimentation you have reference data to compare against. In production you often don't — but you have real users. Design, up front, what user feedback you'll collect, how it correlates with your other metrics, and how you'll feed it back into improvement (the subject of Unit 21).

Annotate — and slice — your evaluation data

Curate annotated examples for every component and criterion, across turn- and task-based evaluation; use real production data where possible. Then slice — evaluate on subsets separately — because aggregate numbers hide problems:

Catch bias	Check performance on minority user groups you'd otherwise average over.
Debug	Poor performance on a subset points at an attribute — length, topic, format.
Find improvements	Weak on long inputs? Maybe a different technique or model.
Avoid Simpson's paradox	A model can win overall yet lose on every subgroup (or vice versa). Only slicing reveals it.

Keep multiple evaluation sets: one matching production distribution for an overall read, plus slices by tier (paid/free), source (mobile/web), known-hard cases, common-typo inputs, and an out-of-scope set to confirm your app declines what it should. The rule of thumb: if you care about something, put a test set on it.

Is your eval set big enough? Bootstrap it An eval set must be large enough to be reliable but small enough to afford. To check 100 examples are enough: resample 100 with replacement repeatedly and re-score. If results swing wildly (90% on one draw, 70% on another), the pipeline isn't trustworthy — you need more data. Useful heuristic for comparing systems: to detect a 3× smaller score difference reliably, you need roughly 10× more samples.

What "good" looks like after this unit

You can now:

Decompose a system and specify how each component is measured — and evaluate per turn and per task, prioritizing task completion.
Write an evaluation guideline: criteria, scope boundaries, scoring rubrics with examples, validated by humans.
Map an eval metric to business impact and set a usefulness threshold, making metric investments defensible.
Choose (and mix) evaluation methods per criterion, and keep humans in the loop in production.
Curate sliced evaluation sets and sanity-check their size by bootstrapping — dodging Simpson's paradox and false confidence.

This is the artifact the capstone (Unit 13) asks you to produce. The next eval unit — 14, the highest-ROI in the series — takes the traces this pipeline logs and turns them into systematic error analysis.