Eval, Testing, and Debugging

Eval, Testing, and Debugging is the lightest domain on the exam at 2.6% — roughly one or two items of 53. Budget accordingly: one careful pass over this material buys the score, and more than that is time stolen from Applications and Integration, which is a third of the paper. Study this domain properly for your job, not for the exam.

The idea that unlocks it: a Claude feature is not tested like a function, it is tested like a distribution. You do not assert that output equals a golden string; you assert that every run satisfies invariants — valid schema, required facts present, forbidden content absent, budget respected — and you measure a pass rate across repeated runs rather than a single result. Cheap deterministic checks run first; a model-based grader only judges what code cannot.

Debugging follows the same discipline. Capture a full trace, reproduce the failure by running it many times to get a flake rate, then isolate the layer — prompt, tool, context, or model — by changing exactly one variable at a time. Every fix earns a permanent case in the eval set.

• Build the eval set from real failures first. Twenty to fifty cases drawn from production traces beat hundreds of invented ones, and every bug you fix becomes a new case.
• Never assert exact string equality on generated prose. Assert invariants: valid JSON, required fields present, a specific tool was called, a forbidden phrase absent, output inside a length or cost budget.
• Run deterministic checks before model-based grading. Schema validation, regex, key-fact presence and tool-call assertions are free, fast and unambiguous; send only what survives them to a judge.
• Lower temperature for tests you want to be stable, but treat it as variance reduction, not determinism — the same request can still produce different text.
• Score a case by pass rate over N runs, not a single run. Pick N and a threshold up front and record both; a flaky-but-passing case is a real signal.
• Grade with a rubric, not vibes. Give the judge the input, the output, explicit criteria and a small ordinal scale, and require a one-line justification before the score.
• Use a cheaper tier as the judge where the task is checkable, and keep the judge prompt frozen — changing the grader invalidates comparison with earlier runs.
• For a prompt change, run the old and new prompt over the same eval set and compare aggregate pass rate plus per-case diffs. A single hand-checked example is not evidence.
• Hold out cases you never tune against. If you iterate on every case you have, you are overfitting the prompt to the eval set.
• Fix tool results in tests. Replaying recorded tool responses makes a multi-turn agent test reproducible and isolates model behavior from a flaky dependency.
• Track cost and latency as first-class assertions, alongside correctness. A prompt change that doubles tokens for a point of accuracy is a regression in most products.
• Version the eval set with the prompt. A pass rate is meaningless without knowing which cases, which grader and which model tier produced it.

• A tool loop that never ends is the classic production failure: the model calls the same tool with the same arguments forever. Cap total turns and total tool calls, and stop the loop from your code — the model is not a reliable stopping condition.
• Detect repeat calls: if identical tool name plus arguments appear twice in a row, the loop is stuck. Return an explicit message saying the call was already made and what the result was, rather than silently re-running it.
• A tool that returns an empty result or a bare error string invites retries. Make the tool result state the outcome in words — no matches found, argument invalid, and what to do next — so the model can move on.
• Intermittent failures are usually context-dependent, not random. Reproduce by replaying the exact conversation prefix, then bisect it — drop or shorten sections until the failure disappears.
• Run a suspected flake many times and record the rate. A bug that appears one run in twenty is a bug; declaring it fixed after two green runs is how it comes back.
• Isolate the layer by changing one variable at a time. Prompt: does a corrected instruction fix it? Tool: does the tool return what you think when called directly? Context: does the fact even appear in the input? Model: does a stronger tier fix it unchanged?
• If the answer is absent from the context, no prompt fix will help — that is a retrieval bug wearing a hallucination costume. Check the input before you touch the prompt.
• Long conversations degrade quietly. Watch for facts buried mid-context, stale tool results contradicting fresh ones, and summaries that dropped the constraint that mattered.
• Log the structure of every turn — model tier, system prompt version, tool names and arguments, stop reason, token counts, latency, error class — and log content by reference or redaction, never raw customer data.
• Redact at the logging boundary, not after the fact. Hash or truncate identifiers, drop free-text fields by default, and keep a short-lived opt-in verbose mode for debugging with real payloads.
• The stop reason is the first thing to read. A truncated answer from hitting the output limit is a different bug from a refusal or a tool-use turn, and they look alike in the rendered text.
• Validate model output at the boundary and fail loudly. Swallowing a malformed response with a try/except and a default value hides the regression until a customer finds it.

TRACE — the order to work a failure, and the order to build a test suite.

• T — Trap the trace. Capture the whole turn before you theorize: system prompt version, full message list, tool calls and their results, stop reason, token counts, latency. A bug you cannot see in a trace is a bug you will guess at.
• R — Reproduce with a rate. Replay the exact input many times and record how often it fails. One failure is an anecdote; a measured flake rate tells you whether you are chasing variance or a deterministic break.
• A — Assert cheaply first. Run deterministic checks — schema, required fields, forbidden strings, tool called, cost and latency budget — before any model-based grading. Most failures are caught here for free.
• C — Classify the layer. Prompt, tool, context, or model. Change exactly one at a time: fix the instruction, call the tool directly, inspect whether the fact is in the context, swap the tier. Two changes at once teaches you nothing.
• E — Enshrine the case. Add the reproduction to the eval set with its expected invariants, so the next prompt change is regression-tested against it forever.

Say it as a sentence: trap it, reproduce it, assert on it, classify it, enshrine it.