AI system design fundamentals

Start simple, add complexity only when the task demands it. The design ladder: (1) a single well‑structured prompt, (2) add retrieval (RAG) when answers depend on facts the model can't have memorised, (3) add tools when it must fetch live data or take actions, (4) add multi‑step agents only when the task needs planning and iteration. Pick the smallest model that clears the quality bar — Haiku for speed and scale, Sonnet as the balanced default, Opus for the hardest reasoning. Every design trades off quality, latency, and cost; name the constraint that matters most before you choose.

The exam loves options that are more machinery than the task needs. Classic traps: jumping to a multi‑agent system when a single well‑structured prompt would do; adding RAG for knowledge the model already has (or that fits in context); fine‑tuning as a first resort instead of prompting + examples; solving a latency problem with a bigger model; and enforcing critical limits only in the prompt. The winning options usually: pick the simplest approach that satisfies the stated constraint, measure before tuning, and put hard rules in code.

• Design ladder: prompt → RAG → tools → agents. Climb only when needed.
• Model tiers: Haiku = fast/cheap/high‑volume · Sonnet = balanced default · Opus = deepest reasoning.
• Add RAG when answers depend on private or fresh knowledge.
• Add tools when the model must fetch live data or act.
• Add agents when the task needs multi‑step planning or decomposition.
• Latency levers: smaller model · streaming · prompt caching · fewer tokens · parallelise.
• Cost levers: smaller model · prompt caching · Batch API · trim context.
• Guardrails first: define what it must *never* do before adding capability.
• Boundaries over trust: enforce limits in code, not just instructions.

• Real‑time chat UX → streaming + the smallest model that clears quality.
• Large offline volume, latency‑insensitive → Batch API: cheaper per request, results asynchronous.
• Repeated static context (system prompt, docs, tools) → prompt caching.
• Long documents → pass only relevant sections (retrieve/summarise) before reaching for max context.
• Images / scans → vision input. PDFs → native PDF support.
• Hardest reasoning → larger model or extended thinking — budget the extra latency.
• Escalation ladder: start with Haiku → measure on your eval set → upgrade only where quality fails.
• Cost = input + output tokens: trim context, cache the static, batch the offline.

“LADDER” — the order to add complexity: Language (a plain prompt) → Augment (RAG) → Do (tools) → Delegate (agents) → Evaluate (measure) → Refine. Most tasks are solved a rung lower than you'd think — don't skip rungs.

“QLC” — pick a model by Quality bar → Latency budget → Cost ceiling, in that order. If two models clear Q, the smaller one wins.

“SCB” — the three cheap wins before switching models: Stream (perceived speed), Cache (repeated context), Batch (offline volume).