Applied GenAI Curriculum for AI PMs · Depth Layer

Unit 21 — Monitoring, Observability & the Feedback Loop

Monitoring and observability for LLM systems, pipeline orchestration, and — the PM's part — user feedback: extracting conversational signals, feedback design, and its limits.

Why this is a spec item, not a launch-day scramble

Unit 10 built the architecture but held two things for launch: how you'll watch the system in production, and how you'll learn from real users. Both should be designed in from the start. Non-determinism (Unit 02) means a system that passed evals can still fail live on the same input, and the conversational interface makes user feedback uniquely rich — and uniquely tricky to read. The takeaway to hold: the feedback loop is part of the product spec.

1. Monitoring & observability

The goal is the same as evaluation — mitigate risks (failures, attacks, drift), find opportunities. The distinction worth knowing: monitoring watches external outputs to notice something's wrong; observability instruments the system so you can figure out what went wrong from its logs and metrics without shipping new code. Three DevOps metrics grade how observable you are:

MTTD

Mean time to detection

How long until you notice something broke?

MTTR

Mean time to response

How long from noticing to resolved?

CFR

Change failure rate

% of deploys that fail and need a fix/rollback. Don't know yours? Redesign for observability.

Design metrics around failures, not the other way round Metrics aren't the goal — catching problems is. So start from what failure you want to catch, then build the metric. Worried about hallucination? Track whether output can be inferred from context. Worried about cost? Track tokens per request and cache hit rate. Evaluation and monitoring must interlock: eval metrics should predict monitoring metrics, and issues found in production must flow back into the eval pipeline (Units 09, 14).

What's worth tracking, at a glance: format failures (invalid JSON — easiest to catch), quality (factual consistency, conciseness via AI judges), safety (toxicity, PII, how often guardrails trigger or the system refuses), latency (TTFT/TPOT/total, per user), and cost (tokens, requests/sec vs. rate limits). Break every metric down by user, release, and prompt version, and combine spot checks (sample) with exhaustive checks (every request). The three observability layers:

Metrics

Aggregated numbers — tell you that something's wrong, at a glance.

Logs

Append-only record of events — tell you what happened. Rule of thumb: log everything (configs, sampling settings, prompt, query, output, tool calls) with tags/IDs, because you can't predict what you'll need.

Traces

Linked events forming one request's full path — let you pinpoint the exact step a query went wrong (bad retrieval? wrong prompt? bad generation?).

Drift — the failure that creeps in silently Three things change under you: the system prompt (a teammate fixes a "typo," a template updates) — add a check that catches when your prompt changes; user behavior (users learn to prompt differently, shifting your metrics gradually); and — most insidious — the underlying model (an API's model gets updated without notice; observed drops of ~10% on a version bump). If you're on someone's API, assume the model can move beneath you and monitor for it.

2. Pipeline orchestration (briefly)

An orchestrator wires the components from Unit 10 into an end-to-end flow — two steps: components definition (declare your models, data sources, tools) and chaining (specify the steps from query to response, e.g. process → retrieve → prompt → generate → evaluate → return-or-escalate). It passes data between steps and flags when the flow breaks.

Don't reach for one on day one Tempting to start with an orchestration framework — but any external tool adds complexity and can abstract away exactly the details you need to debug. Start without one; adopt later, evaluating on integration/extensibility, support for complex pipelines (branching, parallelism, error handling), and ease/performance (avoid ones that make hidden API calls or add latency).

3. User feedback — the PM's real deliverable

This is where you own the design. User feedback does double duty: it evaluates performance and it's proprietary data — the fuel for the data flywheel (Units 03, 11). A product that launches early and gathers feedback builds a moat rivals can't easily close. (And feedback is user data — respect privacy and be transparent about use.)

Explicit vs. implicit

Explicit	Directly requested: thumbs up/down, star ratings, "did we solve your problem?" Easy to interpret, but sparse (users rarely bother) and biased (unhappy users complain more).
Implicit	Inferred from actions. Abundant — "limited only by your imagination" — but noisy and ambiguous (sharing a chat can mean "this is great" or "look at this blunder"). Study your users to learn what each action means.

Conversational signals to mine

The chat interface makes users give feedback the way they would to a person. Signals worth tracking:

From the messages (natural language)

Early termination — user stops the response / leaves Error correction — "No…", "I meant…", rephrasing Complaints — wrong, irrelevant, too long, lacks detail Sentiment — "Ugh," rising frustration "Are you sure?" — distrust or missing detail Refusal rate — "As a language model, I can't…"

From the actions

Regeneration — unsatisfied (or just wants options) Delete / rename convo — delete = bad; rename = good answer, bad title Conversation length — good for companions, bad for support bots Dialogue diversity — long + repetitive = stuck in a loop

The hidden bonus: edits are preference data When a user edits the model's output — fixes generated code, rewrites a draft — that's a gold-standard preference pair: original = "losing," edited = "winning." The same is true for side-by-side "which is better?" comparisons. This is the raw material for preference finetuning (Unit 19), captured for free if you design for it.

Feedback design — when and how to collect

When	What to offer
At the beginning	Calibrate to the user (a language app gauging level). Keep it optional unless calibration is essential (like Face ID); fall back to neutral and learn over time.
When something bad happens	Let users report and still finish the task — downvote, regenerate, edit the wrong output, or escalate to a human. Human-AI collaboration (e.g. image inpainting) both helps the user and yields high-quality feedback.
When the model is unsure	Ask — show two options side by side and let the user pick (great comparative signal), if it doesn't add latency.

How: make it effortless and nonintrusive Feedback must fit the workflow, be easy to give, and easy to ignore. The exemplars bake it into the action itself: Midjourney's upscale/vary/regenerate buttons each carry a different implicit signal; Copilot's Tab-to-accept vs. keep-typing is feedback either way. Two cautions: Apple's guideline warns against soliciting positive feedback (it implies good results are the exception) — though some teams still want it to find beloved features; and gate frequency (ask 1% of users) to avoid clutter, accepting some bias.

The limitation to plan around: integration beats standalone Standalone tools (ChatGPT, Claude) struggle to collect high-quality feedback because they're outside the user's workflow — ChatGPT can't tell if the email it wrote was ever sent. Integrated products (Copilot, Gmail Smart Compose) see whether the output was used or edited. And raw feedback needs context (the surrounding turns) to be actionable — thumbs-down alone tells you the rate, not the reason. Design collection so you capture the context, not just the verdict.

What "good" looks like after this unit

You can now:

Grade observability with MTTD/MTTR/CFR and design metrics backward from the failures you want to catch.
Use metrics → logs → traces to detect, then pinpoint, a production failure — and watch for prompt/behavior/model drift.
Resist reaching for an orchestrator too early, and know what to evaluate when you do.
Design a feedback system: explicit vs. implicit, the conversational signals to mine, and when/how to collect without intruding.
Capture edits and comparisons as preference data — and plan around the standalone-vs-integrated limitation.

That closes the production thread. One DEPTH unit remains before the optional background: defensive prompting — the security lens for any launch exposed to untrusted input.