Applied GenAI Curriculum for AI PMs · Core Sequence · CORE checkpoint next

Unit 12 — Cost & Latency: The Two Constraints That Kill Roadmaps

The inference overview — prefill vs. decode, the standard performance metrics, and where cost actually comes from — so you stop writing roadmap items that are physically impossible at current latency.

Why this closes the CORE pass

You now know how to decide, build, evaluate, and feed an AI system. The last CORE piece is the physics that bounds all of it: every AI feature runs on inference, and inference has a speed and a price you don't get to wish away. You won't optimize inference — that's the engineers' job. But you need enough of the model to know roughly why something is slow or expensive, and to stop writing roadmap items that current latency makes impossible.

1. How generation actually runs: prefill vs. decode

Model inference runs on an inference server (inside a broader inference service that routes and preprocesses requests). If you use an API like OpenAI or Google, that is the inference service; if you self-host, you own it. Generating a response happens in two phases with completely different characteristics:

Compute-bound

Prefill

Reads your input tokens, all in parallel. Limited by raw compute. Its duration scales with input length — this is what you wait through before the first token.

Memory-bandwidth-bound

Decode

Generates output tokens one at a time, each requiring loading large matrices from memory. Limited by how fast data moves, not by compute. This is why longer outputs cost more time.

The one distinction to keep Compute-bound = limited by calculation speed. Memory-bandwidth-bound = limited by how fast data moves between memory and processor. Autoregressive language models are typically memory-bandwidth-bound in decode — which is why "just buy a faster chip" often doesn't help, and why the two phases are frequently run on separate machines. You don't need the roofline math; you need to know that input length and output length pull on different bottlenecks.

2. The metrics that describe "slow"

From the user's side, the axis is latency (response quality is the model's property, not the server's). For streaming responses it breaks into two numbers you'll hear constantly:

TTFT	Time to first token — how fast the first token appears. It's the prefill step, so it grows with input length. Chatbots want this near-instant; document summarization can tolerate more.
TPOT	Time per output token — how fast each subsequent token comes. In streaming it only needs to beat reading speed (~6–8 tokens/sec is fine for most reading).

Total latency = TTFT + TPOT × (number of output tokens)

Two systems with equal total latency can feel very different — snappy first token + slow stream, vs. slower start + fast stream. Which users prefer is a UX decision worth testing.

Two traps in reading latency numbers (1) Averages lie. One 3,000 ms outlier among nine ~100 ms requests makes the "average" 390 ms — misleading. Use percentiles: p50 (median), p90, p95, p99. (2) User-latency ≠ model-latency. For chain-of-thought or agent queries, the model generates hidden intermediate steps first, so the first token the user sees comes much later than the model's own first token. Insist metrics measure time-to-what-the-user-sees.

p50

100 ms

p90

140 ms

p95

210 ms

p99

3,000 ms

Percentiles surface the tail your average hides — the p99 is where a slice of real users actually live.

3. The metrics that describe "expensive"

Throughput — output tokens per second across all users — is what ties to cost. Higher throughput generally means lower cost per token. The cost of a request is simply prefill cost + decode cost, and you can back it out from hardware price and throughput. But raw throughput hides a trap:

The latency–throughput trade, and why "goodput" matters Batching more requests together raises throughput (cuts cost) but hurts latency — it's common to double or triple throughput by sacrificing TTFT/TPOT. So optimizing purely for throughput/cost can quietly wreck the user experience. Goodput fixes this: it counts only the requests per second that actually meet your latency targets (SLO). A service doing 100 requests/min where only 30 hit the latency bar has a goodput of 30 — that's the number that reflects real delivered value.

4. Online vs. batch — the cheapest lever you already have

Providers offer two API modes, and picking the right one is often the biggest cost decision a PM actually controls:

Online API	Optimizes for latency — processes each request on arrival. For anything user-facing: chatbots, code generation. Often offers streaming (tokens shown as generated) to cut perceived wait.
Batch API	Optimizes for cost — trades speed (hours, not seconds) for a big discount (commonly ~50% off). Right for anything without strict latency needs: synthetic data, periodic reports, bulk document onboarding, model migrations, newsletters, reindexing.

The move: before assuming an expensive real-time pipeline, ask whether the workload is actually latency-sensitive. Half of "too expensive" problems are online workloads that could be batch.

The roadmap discipline this unit buys you Because output is generated token-by-token, a long response has a latency floor you cannot prompt your way under. If each token takes ~100 ms, a 1,000-token answer takes ~100 seconds — full stop. Before you scope "instant, comprehensive answers," check whether the token math even allows it. This is the unit that stops physically-impossible roadmap items before they're committed.

What "good" looks like after this unit

You can now:

Explain prefill (compute-bound, input length) vs. decode (memory-bandwidth-bound, output length) at a level that lets you reason about why something's slow.
Read TTFT, TPOT, and total latency — and insist on percentiles over averages and user-visible over model-internal timing.
Connect throughput to cost, spot the latency–throughput trade, and ask for goodput against an SLO.
Route workloads to online vs. batch APIs to control cost without hurting the experiences that need speed.
Sanity-check a roadmap item against the token-by-token latency floor before committing it.

That completes the CORE pass — a standalone operating model from "should we build it?" to "why is it slow?" Next is the checkpoint that proves it produced judgment, not trivia.