Governance, Safety & Risk Management

Governance on CCAR-P is an architecture question wearing a compliance costume. Items describe a deployment — a regulated industry, an agent with tool access, a shared knowledge base — and ask which control actually holds. The trap is answering at the prompt layer: add an instruction, a refusal rule, a policy document, a training session. Instructions are advisory, and a model can be argued out of them by the very content you were worried about. Controls that hold are deterministic and live outside the model: scoped credentials, tool allowlists, schema validation, spend and rate ceilings, an approval gate in front of the irreversible step, append-only logs the agent cannot write to.

The idea that unlocks the domain: treat Claude as an untrusted actor inside your own trust boundary, and treat every token it did not author — retrieved chunks, tool results, user uploads, its own recalled memory — as hostile input. Every governance question then reduces to one placement decision: where does the control point sit relative to the action that cannot be undone.

• Guardrail in code, not in the prompt. An instruction is a preference; a scope, allowlist, schema or quota is a control. Put enforcement where the model cannot argue with it.

• Classify by blast radius before you design. Read-only · reversible write · irreversible write · externally visible. Governance spend belongs to the last two.

• Gate on consequence, not confidence. Human approval sits in front of the irreversible action, not in front of the uncertain one.

• Least privilege is per-agent, per-task. Scope the agent's credential to the smallest resource set that completes the job, and expire it. Shared service accounts destroy attribution.

• Prefer reversibility over review. A staged write with an undo window beats a human approving every action — approval fatigue silently removes the control while leaving the audit record intact.

• Define untrusted content broadly. Anything a user or third party can influence: RAG chunks, web pages, email bodies, file uploads, tool responses, recalled memory. Delimit it, label it, grant it no authority.

• Retrieved text never changes what tools may be called. Tool permissions resolve from the session identity before content is read.

• Enforce authorization at retrieval, not at generation. Filter the index by the caller's entitlements; never ask the model to withhold what you already placed in its context.

• Minimize PII before the boundary. Redact or tokenize on the way in, rehydrate outside the model, so prompts, traces and derived stores hold the least regulated data possible.

• Log the decision, not just the output. Prompt version, model tier, retrieved document IDs, tool calls with arguments and results, approver identity, timestamps — append-only, agent-unwritable.

• Fail closed where it matters. If the policy service, classifier or validator is unavailable, irreversible and externally visible actions do not proceed; degrade to read-only.

• Residency is an architecture constraint, not a setting. Pin inference, retrieval, logs, traces, evals and human review queues together — one component shipping data out of region voids the claim.

• Anti-pattern — the safety prompt. A system-prompt rule as the only defense on a high-blast-radius tool. Any injected instruction competes with it on equal footing.

• Anti-pattern — human-in-the-loop on everything. Reviewers rubber-stamp within days, and the trail then records consent that never happened. Fewer, sharper gates beat universal review.

• Anti-pattern — output filtering as the compliance story. A filter catches what the model said, not what it did; by then the tool call has already fired.

• Anti-pattern — the shared agent identity. One service account for all users means every action carries maximum privilege and no attribution.

• Failure mode — silent success. Tool returns 200, the model reports done, the effect was wrong. Verify the effect, not the response.

• Failure mode — capability creep. Adding one tool moves the agent across a trust boundary nobody re-reviewed. Re-assess the whole permission set when any tool is added.

• Failure mode — confused deputy. The agent spends its own broad privilege on a low-privilege user's request. Propagate end-user identity into every downstream call.

• Failure mode — poisoned memory. Injected instructions persist in summaries, scratchpads and vector stores, then re-fire in a later session. Treat memory as untrusted on read-back; scope it per tenant; expire it.

• Failure mode — retention drift. Transcripts, embeddings, caches and eval sets outlive the policy that only ever covered the primary database. Derived data inherits the sensitivity of its source.

• Regulated deployments differ in evidence, not intent. Named accountable owner, documented intended use and limits, pre-deployment eval results on file, change control over prompts and tools, demonstrable human override. The pattern rarely changes; the paperwork trail does.

• Prompts, tool schemas and retrieval configs are release artifacts. Version them and ship them through the same change control as code — an unversioned prompt cannot be audited.

• Answer with a control point and its layer. "Deny at the tool gateway using the caller's scopes" scores; "add validation and monitoring" does not.

GRAIL — the order to build governance in, from the risk to the record.

• G — Gauge blast radius. Classify every action: read-only, reversible, irreversible, externally visible. Everything downstream is sized from this.

• R — Restrain in code. Turn each risk into a deterministic control — scope, allowlist, schema, quota, kill switch. If it exists only as an instruction, it is not restrained.

• A — Approve at the irreversible step. One gate, in front of the action that cannot be undone, capturing the approver's identity.

• I — Isolate untrusted input and regulated data. Delimit third-party and retrieved content and strip its authority; region-pin and minimize PII at the same boundary.

• L — Log for audit. Append-only provenance — prompt version, model tier, retrieved IDs, tool calls, approver — retained exactly as long as policy allows and no longer.

Use it when a scenario hands you an agent with real-world side effects and asks what to add: walk GRAIL and answer at the first letter the design is missing.