Applied GenAI Curriculum for AI PMs · Depth Layer · Eval Thread 3/5

Unit 14 — Error Analysis & the Improvement Flywheel

Systematic error analysis — open coding, axial coding, criteria drift, strategic sampling — and why looking at your data beats building infrastructure. The highest-ROI unit in the series.

Why this is the highest-ROI unit

The core claim of the entire eval discipline compresses to three words: look at your data. Not dashboards, not generic metrics, not an off-the-shelf eval framework — read your actual production traces and catalog how the product fails. Teams that adopt this consistently find their highest-ROI improvements here, and in real projects 60–80% of development time goes to error analysis and evaluation, not building automated checks. This unit is a DEPTH one, but it's the one to reach for first — ideally right after you've designed the pipeline in Unit 09.

First, what's a trace? A trace is the complete record of a single user session — every message, tool call, retrieval, and response from the first query to the final answer. Error analysis is the practice of reading traces and cataloging failures. Everything below operates on traces.

The error-analysis process

It's a four-step loop borrowed from qualitative research. It is low-tech on purpose — a notebook or a simple custom viewer, a human, and real traces.

Create a dataset

Gather representative traces of real user interactions. No data yet? Generate structured synthetic traces to bootstrap (more on that below).

Open coding

A human reads each trace and writes open-ended notes on what went wrong — like journaling. Focus on the first failure in a trace, since upstream errors cause downstream ones. A domain expert must do this step.

Axial coding ★ most important

Group those raw notes into a failure taxonomy — distinct categories — then count how many traces fall in each. An LLM can help cluster, but you own the categories.

Iterate to saturation

Keep reading traces until you hit theoretical saturation — new traces stop revealing new failure modes. Rule of thumb: review at least 100; if ~20 in a row surface no new category, you can stop.

Open coding — raw notes on individual traces

"Bot gave the return window but never asked if the item was final-sale."

"Prospect asked about pets twice; bot answered once then ignored the repeat."

"Answer was correct but 4 paragraphs for a yes/no question."

↓ axial coding — group & count across 100 traces ↓

Missed clarifying question23

Dropped repeated user intent11

Verbosity on simple queries8

The counts are the payoff: they tell you which failure to fix first. The goal isn't to catalog every possible failure — it's to prioritize the ones that actually happen most.

Criteria drift — why you can't outsource this to an LLM You need criteria to grade outputs — but grading outputs is how you discover your criteria. This is criteria drift: people don't know what "good" means until they see the model's actual outputs. You only learn you need, say, objection-handling after reading a trace where a prospect walked away. A team can't hand a tool context it hasn't discovered yet — which is exactly why looking at data beats delegating evaluation to a generic judge.

Who does it: the benevolent dictator

Use one domain expert who understands your users as the single quality decision-maker — a "benevolent dictator." This matters because error analysis builds the product intuition that only comes from systematically seeing failures. Outsourcing it to people without domain expertise produces superficial or wrong labels and is usually a serious mistake. For a PM, this is often your seat, or one you sit in beside the expert.

Binary labels beat 1–5 scores Label failures as pass/fail, not on a scale. Binary forces a decision instead of hiding uncertainty in a "3," and it's faster — you don't burn the session debating whether something is a 3 or a 4. Write evaluators for the errors you actually found, never the ones you imagine.

The most important investment: a friction-free data viewer

The single biggest enabler is a custom trace viewer that removes all friction from looking at data and renders traces in domain-specific ways — pulling everything you need onto one screen. These can be built in well under a day with a notebook or an AI coding assistant. If reading a trace is annoying, the team won't do it, and the whole flywheel stalls.

Bootstrapping data when you have none

No production traffic yet? Generate synthetic traces — but structured, not "give me test queries" (which yields generic mush). Define dimensions of user variation, then combine them:

1. Define dimensions	Axes of variation — e.g. for support: Issue Type × Customer Mood × Prior Context.
2. Write ~20 tuples by hand	Specific combos like `(billing, frustrated, follow-up)` — this teaches you the problem space.
3. Scale in two steps	LLM generates more tuples, then a separate prompt turns each into natural language (avoids repetitive phrasing).
4. Run & sample	Push queries through the real system, capture traces, sample ~100 for error analysis.

Two cautions: fix obvious problems directly rather than generating tests for them, and distrust synthetic data in complex/high-stakes/low-resource domains where it misses the real edge cases.

Making it a flywheel — and selling it

Error analysis isn't one-and-done. Re-run it whenever you make significant changes (new feature, prompt update, model switch, big bug fix), aiming for 100+ fresh traces per cycle (typically every 2–4 weeks), with 10–20 outlier traces reviewed weekly in between. Over time you develop a "nose" for where failures hide, and strategic sampling (outliers, user-feedback sorting, clustering) makes each pass sharper.

How to sell evals to a skeptical team: don't Don't pitch "evals." Do the error analysis yourself on 50–100 real conversations, then tell a story with the data — the top failure modes, how often the costly ones happen, surprising user behavior, and bugs you already fixed (framed as "prevented production issues"). Show the error rate for a specific problem dropping after your fix. Let results lead; the team will ask how you did it. And measure your roadmap in experiments run, not features shipped.

Run a 30-minute error-analysis session

Pull ~20–50 recent real traces into a friction-free viewer.
Open-code: for each, jot the first thing that went wrong (binary: acceptable / not).
Axial-code: cluster your notes into 4–8 failure categories and count them.
Rank by frequency × impact — the top one or two are your next work.
Fix the obvious bugs immediately; write an eval only for failures worth automating.
Note what surprised you, and schedule the next pass (2–4 weeks or after the next big change).

What "good" looks like after this unit

You can now:

Run the open → axial → iterate loop on real traces and stop at theoretical saturation (~100 traces / 20-no-new heuristic).
Explain criteria drift — and why it means evaluation can't be fully outsourced to a generic LLM judge.
Set up a benevolent dictator, binary labels, and a friction-free data viewer.
Bootstrap with structured (dimension-based) synthetic data when you lack traffic — and know when not to trust it.
Sell the practice by telling a story with data, and run a 30-minute session start to finish.

Error analysis surfaces failures; some you fix directly, others you automate with an evaluator. When that evaluator is an LLM judge, it has to be validated — which is exactly the next unit.