Applied GenAI Curriculum for AI PMs · Depth Layer · Optional Background

Unit 24 — Foundation-Model Evaluation Concepts

Why foundation models are uniquely hard to evaluate, exact evaluation, and comparative / arena-style ranking — enough to parse vendor benchmark claims critically.

Why this optional unit exists

The eval thread (06 → 16) was about evaluating your product. This unit is about evaluating the models themselves — the vocabulary researchers and vendors use when they say "we top the leaderboard" or "89% on MMLU." You can build great products without it, but when a vendor waves a benchmark number, this is what lets you ask the right skeptical question instead of nodding along. It complements, from the research side, the "why public benchmarks mislead" material back in Unit 07.

1. Why foundation models are uniquely hard to evaluate

Evaluating any ML model is hard; foundation models are harder, for reasons worth naming:

Smarter = harder to grade

Anyone can mark a first-grader's math; few can grade a PhD proof. Judging a coherent-but-possibly-wrong summary may require reading the whole book first.

Open-ended breaks ground truth

Classic ML is close-ended (output vs. expected category). Open-ended tasks have countless valid answers — you can't curate a complete list to compare against.

Black boxes

Providers often hide architecture, data, and training. Without those, you can only judge a model by its outputs.

Benchmarks saturate fast

Models hit perfect scores and the benchmark dies — GLUE → SuperGLUE, MMLU → MMLU-Pro. A leaderboard win ages quickly.

A fifth wrinkle: general-purpose models mean evaluation isn't just measuring known tasks but discovering what the model can do. And bluntly — evaluation is under-invested relative to modeling, so infrastructure is thin and a lot of teams still "just eyeball it."

The frame this sets up Every hard part above is a reason a single benchmark number can't capture a model's real quality. So when you see one, the reflex is: hard at what, judged how, and does it match my task?

2. Exact evaluation — when a score is unambiguous

Exact evaluation produces judgment without ambiguity (pick B when the answer is A → wrong). Contrast with subjective evaluation (essay grading, AI-as-judge) where the score shifts with the grader. Two exact approaches:

Functional correctness

Did it actually do the job? For code, run it: does gcd(15,20) return 5? Automatable for anything with a measurable objective — code (HumanEval), text-to-SQL, game bots, optimization tasks. The ultimate metric, but not always automatable.

Similarity vs. reference data

Compare the output to a known-correct answer (input → reference response). Human references treat human performance as the gold standard. Bottlenecked by how fast you can produce references — increasingly AI-generated, then human-reviewed.

Decode "pass@k" — the metric behind most coding claims A model generates k code samples per problem; it "solves" the problem if any of them passes all tests. pass@k is the fraction solved. The catch: more samples = more chances, so pass@10 > pass@3 > pass@1 for the same model. A vendor quoting "pass@10" is a weaker claim than the same number at pass@1 — always check the k.

For similarity metrics, "reference-based" needs a gold answer; "reference-free" doesn't (usable in production where there's no ground truth). And a recurring trap: low-quality reference data poisons the score — a correct translation can be marked wrong just because it wasn't in the reference set.

3. Comparative / arena-style ranking

Often you don't care about a model's absolute score — you want to know which is best. Two ways to rank:

Pointwise	Score each model alone, then sort. Like scoring each dancer individually.
Comparative	Put models head-to-head and ask which wins. Easier for subjective quality — telling which of two songs is better beats scoring each. This powers arena leaderboards (Chatbot Arena) and the "which response is better?" prompts you've seen in ChatGPT.

Each head-to-head is a match; the fraction A beats B is A's win rate. With many models, a rating algorithm (Elo, Bradley–Terry, TrueSkill — borrowed from chess and games) turns match histories into a ranking. Key insight: a ranking is really a prediction — there's no ground-truth "correct" ranking; a good one predicts future match outcomes well.

The three ways arena rankings can mislead Scalability: pairs grow quadratically, so even hundreds of thousands of votes average out to thin coverage per pair (and per task). Transitivity may not hold: rankings assume A>B and B>C implies A>C, but human preference isn't always transitive. No standardization: crowd voters use any prompt with no shared bar — they may prefer answers that sound better but are factually wrong, and "preference in the wild" may not fit your use case (a model that refuses an inappropriate joke gets downvoted).

The pitfall that matters most to a PM: preference ≠ correctness Not every question should be settled by preference. "Is there a link between cell-phone radiation and brain tumors?" has a correct answer, not a preferred one — and preference voting can train misaligned behavior. Comparative evaluation only works when voters actually know the subject — i.e. AI-as-assistant on tasks users can already do, not AI answering things users don't know. When a vendor cites arena wins, ask whether the questions were preference-appropriate or correctness questions dressed up as preferences.

One more distinction to keep straight: comparative evaluation (two outputs shown at once, pick the better) is not A/B testing (one output per user, measure downstream behavior — Unit 06's Level 3).

What "good" looks like after this unit

You can now:

Explain why a single benchmark number can't capture a foundation model's quality (open-endedness, saturation, black boxes).
Read exact-evaluation claims — decode pass@k (and check the k), and spot reference-data quality as a failure point.
Understand arena/comparative rankings as predictions, and name their three weaknesses (scale, transitivity, standardization).
Apply the preference-vs-correctness test to any leaderboard claim — the single sharpest question to ask a vendor.

That completes all 24 units. The series' shared reference list — every source, for further reading — follows as the closing page.