Applied GenAI Curriculum for AI PMs · Depth Layer · Eval Thread 4/5
When AI-graded evals can be trusted: binary pass/fail over Likert scores, validating the judge against human labels, and the judge's own biases and limits.
Sooner or later a team says "let's have an LLM grade the outputs so we can scale evaluation." That's a fine instinct — human review doesn't scale — but an unvalidated LLM judge is worse than no metric, because it launders guesses into official-looking numbers. This unit gives you the two things a PM needs: how a trustworthy judge is actually built, and how to tell a calibrated one from a decorative one. The rule you leave with: never ship a decision on a judge that hasn't been validated against human labels.
The most common mistake is a dashboard of metrics scored 1–5 by an LLM. It looks rigorous; it's usually noise. Start binary.
The gap between 3 and 4 is subjective and inconsistent across graders. Annotators hide uncertainty in the middle. Detecting a real difference needs bigger samples. Worst: these scores usually don't correlate with what the domain expert actually cares about.
Forces a real decision — "passed because…", "failed because…". Faster to label, clearer to act on, and it makes the grader articulate what actually matters. Track gradual progress by splitting into multiple binary checks (e.g. "4 of 5 expected facts present"), not a scale.
The method for producing a judge you can actually trust. Note it starts and ends with a human — the LLM judge is the output, not the starting point.
One person whose judgment defines "good" for this product (a lawyer for legal AI, a support director for a support bot). Not a convenient proxy — one authoritative voice, or evaluations fracture into conflicting opinions.
Cover the interactions you'll see in production, structured by dimensions that fit your case (e.g. features × scenarios × personas). Real data is best; structured synthetic user inputs fill gaps.
One question: "did the AI achieve the desired outcome?" Plus a written critique explaining why. The critique is the gold — it forces the expert to externalize unspoken criteria.
Error analysis will surface plain bugs — fix them now rather than building a judge to detect what you can just repair. Re-stabilize, then continue.
Write a judge prompt using the expert's critiques as few-shot examples (each: input, output, critique, pass/fail). You can't write a good judge prompt until you've seen the data — criteria drift again.
Compare judge vs. expert on fresh examples, refine the prompt, repeat until they agree. Then apply the judge at scale and compute error rates by dimension.
Critiques must be detailed enough to drop into a few-shot judge prompt — "a new employee could follow it." Terse critiques are the most common judge-prompt mistake.
A judge is only worth its agreement with the human expert. Track alignment as you iterate; the Honeycomb example reached >90% agreement in three iterations. But how you measure alignment matters:
An unexpected bonus: building the judge makes the expert sharper. Reading the LLM's critiques, they realize where their own judgments were inconsistent — the process standardizes the criteria and surfaces new product insight, often worth more than the judge itself.
An AI judge is still an AI application, so it inherits every weakness in this curriculum — plus some specific to judging. Know these so you interpret (and discount) scores correctly:
| Inconsistency | Same judge, same input, different score across runs — it's probabilistic. Examples in the prompt raise consistency, but consistent ≠ accurate (it may consistently repeat a mistake), and longer prompts cost more. |
| Criteria ambiguity | Judge metrics aren't standardized — "faithfulness" means different prompts and scales in different tools. Two "faithfulness" scores aren't comparable. |
| Cost & latency | A judge in the request path adds latency and cost; evaluating before returning to users trades risk for wait — sometimes a nonstarter. |
| Bias | What it does |
|---|---|
| Self-bias | A model favors its own outputs (one model rated itself ~25% higher). Be wary of a judge grading its own family. |
| Position / first bias | In pairwise comparisons it tends to favor the first option. Mitigate by re-running with orders swapped. (Humans have the opposite — recency bias.) |
| Verbosity bias | Prefers longer answers regardless of quality — sometimes even a longer wrong answer over a short correct one. Fades with stronger models. |
Which model should judge? Stronger, weaker, or the same — each has trade-offs. A common, cost-smart pattern: a cheap model generates all responses, a strong model judges just a 1% sample. Two catches: the strongest model has no eligible judge, and self-evaluation (a model judging itself) is good for sanity checks but suspect for scoring, thanks to self-bias. Whatever you pick, the judge should supplement exact metrics and human review — never fully replace them.
That's the last method unit of the eval thread. The final piece is a reference shelf for the specific questions that come up mid-project — Unit 16.