Evaluation, Testing & Optimization

Evaluation is 16% of CCAR-P and the domain where implementation instincts fail hardest. Questions rarely ask how to compute a metric. They ask which metric a given business objective demands, when an offline score stops predicting production behavior, and how to ship a prompt or model change without betting all traffic on it at once.

The trap is treating evals as QA — a suite written after the system works, scored on examples the team happened to remember. That set is drawn from the developer's imagination rather than from production, so it passes everything and predicts nothing. The second trap is trusting an LLM-as-judge nobody validated: judges reward verbosity, favor whichever answer sits first, prefer their own outputs, and drift silently when the judge model or judge prompt changes underneath you.

The idea that unlocks the domain: an eval set is a design artifact, built before the system, from real traffic and known failure cases, with the decision each score must inform written down first. Offline gates, shadow runs, canaries, A/B tests and cost·quality·latency tuning are all downstream of owning a scorer you trust.

• Start from the decision, not the metric. Write down what you will do if the score drops, then pick the metric that triggers that action. At least one metric should map to money or time saved.

• Build the eval set before the system. Seed it from real or realistic traffic, stratify by intent, difficulty and user segment, and keep the hard tail over-represented.

• Every production failure becomes a permanent test case. The suite is an asset that compounds; incidents are its main source of new data.

• Reach for scorers cheapest-first: deterministic checks (schema, tool-call validity, regex) · classical metrics (exact match, F1, recall@k) · LLM-as-judge for open-ended quality · human review for the ambiguous tail.

• Offline answers "did this break something known?" and gates the release. Online answers "does this help real users?" and decides promotion. Neither substitutes for the other.

• Grade RAG stage by stage: retrieval on recall@k and precision, generation on groundedness, faithfulness and answer relevance. One end-to-end number hides which half is broken.

• Score agents on the trajectory, not just the final answer: task completion, tool-call validity, step count, recovery after an error, and cost per completed task.

• Pin every run — dataset version, prompt version, judge model, judge prompt, temperature at 0 where determinism is wanted. Change one variable per run or the result is uninterpretable.

• Prefer discrete rubric levels and pairwise A-vs-B comparisons over absolute 1–10 scores; give the judge a reference answer whenever one exists.

• Optimization order: prompt and context first (free) · model tier routing next (small tier by default, escalate on complexity or low confidence) · then prompt caching and batch processing · fine-tuning last.

• Treat cost and latency as first-class eval outputs. Track cost per resolved task and p95 latency, not tokens per call — a cheap call that needs three retries is not cheap.

• Set the pass bar per use case. A summarizer can ship at 85% acceptable; an agent that issues refunds needs a near-zero rate on the one failure class that costs money.

• Imagined eval set. Examples invented by the team pass at 100% and predict nothing. Sample from production logs and slice by intent, difficulty, segment and locale.

• Leakage. Examples used to tune prompts silently become the score. Hold out a set nobody optimizes against and rotate it once it starts saturating.

• Unvalidated judge. If you have not measured the judge's agreement with human labels on a sample, you do not know whether it is scoring correctness or style.

• Judge biases to name: verbosity (longer reads as better), position (first or last option favored), self-preference (a model rates its own output higher). Counter with length-controlled rubrics, randomized order with swapped repeats, and a judge from a different family or tier than the generator.

• Averaging away the failure that matters. 4% error concentrated in the highest-value segment is a different product from 4% spread evenly. Always report slices alongside the headline number.

• Goodhart's law. A metric under optimization pressure stops measuring quality. Pair every target metric with guardrails — helpfulness with hallucination rate, resolution rate with escalation and refusal rates.

• Non-determinism read as regression. Sample N times per item and compare distributions; a single-sample diff between two runs is noise, not a finding.

• Shipping straight into an A/B test. Sequence it: offline suite → shadow (production traffic, output logged not served) → canary slice → full A/B.

• Peeking. Stopping an experiment the moment it looks good inflates false positives. Fix the sample size up front or use a sequential test, and pre-register the primary metric.

• Proxy mistaken for outcome. Thumbs-up rate is not deflection; a similarity score is not usefulness. Validate the proxy against the business metric once before you rely on it.

• Silent quality loss from cost tuning. Any context trim, tier downgrade or cache change must be re-scored on the same eval set before it counts as a win.

• No monitoring means no evaluation. Log inputs, outputs, scores, cost and latency; alert on input distribution shift, not only on error rate — drift shows up upstream of failures.

PROVE — the order to build and run an evaluation program.

• P — Pin the objective. Name the business decision the score must drive, and the cost of each failure type, before choosing any metric.

• R — Representative set. Sample real traffic, stratify by intent, difficulty and segment, hold out a clean split, and keep every production failure as a permanent case.

• O — Objective scorers first. Deterministic checks and classical metrics before any LLM judge; the judge only grades what code cannot.

• V — Validate the judge. Measure agreement with human labels, randomize order, control for length, and version the judge model and prompt.

• E — Experiment online. Shadow, then canary, then A/B with a pre-registered primary metric, guardrails and a stopping rule — and keep monitoring after rollout.

Use PROVE when a question asks how to evaluate something, whether a score can be trusted, or in what order to ship a prompt or model change.