Applied GenAI Curriculum for AI PMs · Depth Layer · Eval Thread 5/5
The FAQ distilled from teaching hundreds of engineers and PMs — plus the sharp idea that product design and evaluability are the same problem. A lookup, not a linear read.
The other eval units teach a method start-to-finish. This one is a shelf you return to when a specific question comes up: should I buy an off-the-shelf metric? build a tool? write an eval for this? put a check inline? Skim it now so you know what's here; come back when you need the answer. It closes with the single most useful reframe in the whole thread — that a product being "hard to eval" is usually a product problem, not an eval problem.
Should I use ready-to-use / generic metrics? No
Generic scores — helpfulness, coherence, quality, BERTScore, ROUGE, cosine similarity — measure abstract qualities that may not matter for your task. Good scores don't mean your system works; they mostly create false confidence. They won't catch that a real-estate bot suggested an unavailable showing (an assertion catches that) or confused two client personas (a validated judge catches that).
The one legitimate use: as exploration signals to surface interesting traces worth reading — never as the measure of quality itself.
Should I build my own annotation / trace-viewing tool? Usually yes
The highest-leverage investment is a friction-free, domain-specific viewer that puts everything you need to judge a trace on one screen. Off-the-shelf dashboards nudge you toward generic metrics; a custom viewer (buildable in under a day with a notebook or AI coding assistant) keeps the team actually looking at data. Don't buy fancy tools before you've used what you have.
Should I practice eval-driven development (write evals before features)? Generally no
Unlike normal software, LLMs have infinite failure surface — you can't anticipate what breaks. Writing evaluators for imagined failures wastes effort and blocks you. Start with error analysis; write evals for failures you actually discover. Narrow exception: a crisp, known constraint like "never mention competitors" can be written up front.
Should I build an automated evaluator for every failure I find? No
First fix obvious gaps (you often just never specified "be concise" or a format). Then apply the cost hierarchy: cheap code checks (regex, schema, execution tests) for anything rule-shaped; an LLM judge only for subjective qualities and only for problems you'll iterate on repeatedly — because a judge costs 100+ labeled examples plus ongoing weekly maintenance and cross-team coordination. Always run a cost-benefit check: does this failure justify the investment?
How much should I fixate on model selection? Less than you think
Swapping models is the reflex many teams reach for first. Don't — without evidence. Do error analysis and ask whether the data actually says the model is the problem. Usually the fix is context, prompt, or a bug, not a new model.
These are complementary systems; the key difference is the data they run on.
| CI / CD | Production | |
|---|---|---|
| Dataset | Small, curated (~100+): core features, regression tests for past bugs, known edge cases. | Sampled live traces — usually no reference answer available. |
| Methods | Favor deterministic assertions (run often, cost matters). | Lean on reference-free LLM judges, run async; track confidence intervals. |
| Trigger | Every change. | Continuous sampling; alert if the CI lower-bound crosses threshold. |
The loop that ties them: when production error analysis reveals a new failure pattern, add representative examples to the CI set so it can't regress.
Fast, deterministic, explainable (regex, schema, block-lists, light classifiers). Target clear-cut, high-impact failures (PII, injection, malformed JSON). A false positive is a production bug, so keep them conservative. You'd almost never put a slow LLM judge here.
Measure nuanced qualities rules can't (factual correctness, completeness). Feed dashboards and regression tests; they don't block the answer. Heavier compute (LLM judge) is fine because they run in the background.
An evaluator can double as a guardrail only if it's fast/cheap enough for the request path and the false-positive/false-negative trade favors blocking — high-stakes domains may accept blocking, creative ones may not.
The most common objection to evals is "our product is hard to evaluate." That's usually a product problem, not an eval problem: an output that's hard for you to verify is hard for your users too — in the worst case they must redo the work from scratch to trust it. Designing for verifiability comes before building evals.
"Net revenue was $4.21M." Nothing to check — the user must redo the analysis to trust it. A 50-page report the doctor must re-read the whole chart to verify.
Show provenance (which vetted source, who authored it), the assumptions made, intermediate numbers, and a flagged list of what couldn't be verified — each as a scoped unit to accept, edit, or reject.
That completes the eval thread. From here the remaining DEPTH units cover architecture internals, data, monitoring, and safety — pulled in whenever a live decision calls for them.