Applied GenAI Curriculum for AI PMs · Depth Layer · Architecture Thread

Unit 18 — Agent Internals: Tools, Planning, Failure Modes, Memory

Tool design, planning approaches, the agent failure-mode taxonomy, and memory systems — enough to contribute to agent scoping, especially the failure-mode budgeting where agent projects live or die.

Why open up the agent

Unit 04 taught you to resist agents until they're justified. This unit is for the moment one clears the bar and gets greenlit — because "we're building an agent" is not one decision, it's four: what tools it gets, how it plans, how you'll catch its failures, and how it remembers. The single most important of these for a PM is failure-mode budgeting — agents fail in more ways than any other system, and projects live or die on whether you planned for that.

1. Tools — what the agent can actually do

An agent's capabilities are defined by its tool inventory. Tools fall into three categories, escalating in power and risk:

Knowledge augmentation

Read-only info gathering — retrievers, SQL executors, web/search APIs, email readers. Keeps the model current and grounded.

Capability extension

Fixes inherent weaknesses — calculator, code interpreter, unit/timezone converters, or another model (image gen, OCR, transcription) to add a modality.

Write actions

Change the world — send email, place order, update a database, initiate a transfer. Power and danger live here.

Write actions are the risk line Read-only tools make an agent smarter; write actions make it consequential. "You shouldn't give an intern authority to delete the production database" — same for an unreliable agent initiating transfers. For any write action, the PM question is: what's the blast radius, what verifies it before it fires, and does it need explicit human approval? Harm isn't only physical — an agent can move markets, leak data, or spread misinformation without ever touching the world.

Choosing tools is empirical, not a checklist. More tools = more capability but harder use (and longer descriptions that eat context). The moves: compare agent performance across tool sets; run an ablation (drop a tool — if performance holds, remove it); find tools the agent keeps misusing and simplify or swap them. Note different tasks and even different models prefer different tools.

2. Planning — how the agent decides what to do

A task is a goal plus constraints ("two-week SF→India trip" is the goal; "$5,000 budget" is the constraint). A plan is the roadmap of steps. The critical design choice:

Decouple planning from execution You can let a model plan and execute in one shot — but then a bad 1,000-step plan runs for hours burning API money before anyone notices. Instead: generate a plan → validate it (cheap heuristics like "reject plans using tools it doesn't have" or "reject plans over X steps," or an AI judge) → only execute validated plans → reflect on the outcome. This is the single most important lever for controlling agent cost and risk.

Generate plan → Validate → Execute → Reflect / correct ↺

Reflection (evaluate the plan, then evaluate the results — "did this actually accomplish the goal?") isn't mandatory but sharply improves performance. Humans can sit at any stage: provide a high-level plan, approve a risky step, or execute sensitive operations — so define the level of automation allowed per action.

Can models even plan? (the honest caveat) It's genuinely contested. Planning is a search problem (explore paths, predict outcomes, sometimes backtrack), and some argue autoregressive models can't truly do it — that plans "look reasonable to a lay user yet fail at execution time." Others counter the models just lack the right tooling and can effectively backtrack by revising. The PM takeaway: don't assume the plan is sound because it reads well. That skepticism is exactly why validation and reflection exist.

3. The failure-mode taxonomy — the heart of this unit

Evaluation is detecting failures, and agents have more failure surface than anything else. To evaluate an agent you identify its failure modes and measure how often each happens. Three families:

Planning failures

Tool-use errors: invalid tool (calls one it doesn't have); valid tool, wrong number of parameters; valid tool, wrong parameter values.
Goal failure: doesn't solve the task, or solves it while breaking a constraint (plans the trip but blows the budget).
Time: an often-overlooked constraint — a grant proposal finished after the deadline is worthless.
Reflection error: the agent insists it finished when it didn't (assigns 40 of 50 people to rooms and declares success).

Tool failures

Right tool, wrong output (captioner mis-describes; SQL generator returns a wrong query).
Translation errors turning a high-level action into an executable command.
Missing tool — the task needs the internet and the agent has no web access. Test each tool independently; print every tool call and output.

Efficiency failures

A valid plan with the right tools that's still too slow or too expensive. Track average steps, cost, and time per task against a baseline (another agent or a human) — remembering "efficient" differs for humans vs. AI.

Failure-mode budgeting — where agent projects live or die Before greenlighting, don't ask "will it work?" — ask "which of these failure modes can we tolerate, at what rate, and what catches each one?" Concretely: build a planning dataset of (task, tools) pairs, generate K plans per task, and measure — what fraction of plans are valid? how many tries to get a valid plan? what fraction of tool calls are valid? Those numbers are your budget. An agent shipped without them is an agent whose failure rate you'll discover in production.

4. Memory — what the agent retains

Agents juggle instructions, context, tool outputs, plans, and reflections — usually more than fits in context. A memory system supplements it. Three mechanisms, mirroring human memory:

Internal knowledge

What the model learned in training — always available, changes only if you retrain/finetune. (Like knowing how to breathe.)

Short-term memory (context)

The conversation so far — fast, but capacity-limited by context length; holds what's most relevant right now. (The name of someone you just met.)

Long-term memory (external)

External sources fetched via retrieval — persists across tasks, cheap to extend, deletable without retraining. (Your books and notes.)

Which to use depends on frequency: universal info → internal knowledge (training); rarely-needed → long-term; immediate → short-term. Memory buys real things: handling info overflow within a session, persisting preferences between sessions (an AI coach you don't re-explain your life to each time), boosting consistency, and preserving structure (store leads in a sheet, actions in a queue).

The hard part is managing limited short-term memory — deciding what to keep vs. drop. FIFO (drop the oldest) is simple but dangerous — the earliest message often states the whole purpose. Better strategies remove redundancy via running summaries, entity tracking, or reflection that decides whether new info should be added, merged, or replace outdated (contradictory) memory.

What "good" looks like after this unit

You can now:

Classify an agent's tools (knowledge / capability / write) and treat write actions as the risk line needing approval and blast-radius limits.
Insist on decoupled plan → validate → execute → reflect, and stay skeptical of plans that merely read well.
Name the agent failure taxonomy (planning, tool, efficiency) and run failure-mode budgeting with concrete plan/tool-validity metrics before greenlight.
Reason about the memory hierarchy and the keep-vs-drop problem (FIFO's danger, summary/reflection strategies).

That's the agent deep dive. The last architecture-thread deep dive is finetuning — Unit 19 — for when someone seriously proposes changing model weights.