AI Agent Architecture — A Step-by-Step Guide¶

A self-contained guide to how AI agents are built, organised as a progression from concepts to concrete mechanisms. Read top-to-bottom on a first pass — each section assumes the previous ones.

A note on terminology

The word "agent" carries a lot of marketing weight. This guide uses it broadly: an agent is a software system that uses a language model as a reasoning engine to take actions in the world via tools. Within that broad category there's a critical distinction between workflows and autonomous agents — covered in §2 — and the distinction matters more than the label.

Table of contents¶

Foundations: what an agent actually is
The loop: workflows versus autonomous agents
Anatomy of an agent
The stateless reality of LLM APIs
How agents get activated
Memory, in depth
Context engineering: what goes in the window
Observability: tracing, evals, and human-in-the-loop
Code execution: how an agent runs code
Sandbox architectures
Integration: MCP as the connective standard
The build-options landscape
Cost architecture: the economics of viability
Pitfalls and how to avoid them
A reference architecture
Glossary

1. Foundations¶

1.1 The mental model: an LLM is a function¶

A language model, in pure form, is a stateless function: text in, text out. (More precisely, context in, tokens out — modern models also accept images, audio, and structured documents — but for reasoning about architecture, treat it as a stateless function.) On each call, it sees the input you provide and produces a response. It remembers nothing between calls. It cannot do anything by itself — no file system, no database, no network, no clock. It can only read text and write text.

Everything else you've heard about AI agents is scaffolding around this function. Once you internalise that, the architecture stops feeling magical and starts feeling tractable.

1.2 The "augmented LLM"¶

A useful agent is built by augmenting that pure function with three capabilities:

Tools — external functions the model can decide to call. A database query, an API request, a file write, a calculator. The model doesn't execute these; it emits a structured request and something else executes on its behalf.
Memory — context across time. Short-term (the running conversation) and long-term (facts, summaries, retrieved documents).
Retrieval — pulling relevant external information into the prompt before the model reasons. The "R" in RAG.

This trio is called the augmented LLM. An agent is built on top of it by wrapping it in a loop.

1.3 What separates an agent from a chatbot¶

A chatbot takes a turn and answers. An agent takes a turn, reasons, calls tools, observes results, reasons again, and keeps going until the task is done. The architectural difference is not the model — it's the loop and the tools. Strip the loop and tools away and an agent collapses back into a chatbot.

2. The loop¶

2.1 ReAct: the canonical agent loop¶

The dominant pattern for agent loops is ReAct — "Reasoning + Acting". The model alternates between thinking about what to do and doing it:

loop:
    1. Model reads the current state (conversation + tool results so far)
    2. Model emits either:
         (a) a tool call  → execute it, append the result, loop
         (b) a final answer → exit

That's the entire algorithm. It is deceptively simple. Most real agents are ReAct with extra structure layered on top.

2.2 Workflows versus autonomous agents¶

Inside this loop, you have a critical architectural choice: who decides which tool to call and in what order?

In a workflow, that decision is made by your code. You hand-wire the steps: "first extract the entities, then query the database, then summarise." The LLM is called at each step to do a specific sub-task, but the orchestration is deterministic and reviewable.
In an autonomous agent, that decision is made by the LLM itself. You give it a goal and a set of tools and let it figure out the sequence. Flexible, but unpredictable, costlier, and harder to debug.

The most important insight in this area: most production "AI agents" are actually workflows. Autonomous agents are appropriate when you genuinely cannot script the path in advance — and they're rarer than the hype suggests.

A practical rule: start with a workflow. Move toward agent autonomy only when the workflow becomes unmanageable.

2.3 The five workflow patterns¶

Most useful agent-like systems compose from a small set of patterns. Knowing them by name is half the battle.

Prompt chaining — Output of step 1 is input to step 2. Each step is a separate LLM call with its own prompt. Useful when a task decomposes naturally: parse → analyse → summarise.
Routing — An LLM classifies the incoming request, and your code sends it to the appropriate specialised flow. "Is this a billing question, a tech-support question, or a sales question?" → route.
Parallelisation — Run the same or different prompts in parallel over the same input, then combine. Used for cross-checking ("ask three times, take the majority"), or for independent sub-tasks ("review this contract for legal, financial, and operational concerns simultaneously").
Orchestrator–workers — A coordinator LLM breaks a complex task into sub-tasks and farms them out to worker LLMs (or worker agents). It then integrates the results. Useful for research-style tasks.
Evaluator–optimiser — One LLM produces output; another LLM judges it; the first revises based on feedback. Iterate until the judge is satisfied. Good for tasks where quality matters more than latency.

True autonomous agents are essentially "loop until done" wrapped around ReAct. They subsume all five patterns but at the cost of predictability.

A corollary: be sceptical of multi-agent

Most problems that look like they need a team of agents are more reliable as a single agent plus a workflow. Every agent-to-agent handoff is a lossy context transfer, and multiple autonomous agents compound non-determinism and make failures hard to trace. Reach for multi-agent only when sub-tasks are genuinely independent and parallelisable — fan-out research with a single synthesiser, say — and even then, keep the orchestration in your code, not in another agent's discretion.

Contested: ai-adoption-maturity-model argues the opposite ceiling

That page (from Anthropic's Steps of AI Adoption, July 2026) treats scaling agent count as the adoption path itself — step 3 is an "org tree" of subagents (~100), step 4 has agents kicking off agents (~1,000+), with orchestration explicitly in the agent's discretion. Much of it describes independent tasks under worktree isolation, which is the case this warning already permits; the unresolved part is hierarchical delegation. The underlying disagreement is whether lossy handoff is intrinsic to agent-to-agent transfer (this guide) or an engineering problem dissolved by verification loops, sandboxing, and encoded standards (that page). Both agree verification must precede autonomy — "bounded autonomy" here and "trust in the loop" there are the same variable.

3. Anatomy¶

An agent is composed of distinct parts, often confused with one another. Naming them precisely prevents most of the confusion.

3.1 The model¶

The language model itself. A black box that takes a context window of messages plus tool definitions, and returns either a text response or a structured tool-call request.

3.2 The tools¶

External functions the model can call. Each tool has:

A name — the identifier the model uses to call it.
A description — a natural-language explanation of what the tool does and when to use it.
A schema — a typed specification of the arguments (typically JSON Schema).
An implementation — actual code that executes when the tool is called.

The quality of the description and schema directly determines how well the model can decide when and how to use the tool. Vague descriptions produce unreliable agents.

3.3 The harness¶

This is the most important word in agent engineering, and the most under-discussed. The harness is the code that sits between the model and everything else. Its job is to:

Expose the available tools to the model (generate the schema, send it in each request).
Receive the model's response.
If the response contains a tool call, dispatch it to the corresponding implementation.
Format the result and feed it back into the next model call.
Loop until the model produces a final answer.

A harness is small — a basic version is fifty lines of code. Frameworks like Microsoft Agent Framework, LangChain, LangGraph, OpenAI Agents SDK, and many others are harnesses, with extra features (streaming, memory, tracing, multi-agent orchestration) layered on.

Crucially, the model never executes anything directly. It emits a structured intent ("call this tool with these arguments"); the harness executes. This separation is what makes agents safe to reason about.

3.4 The context¶

The set of messages and other data sent to the model on each call. This is the only thing the model sees. If something isn't in the context, it might as well not exist. The whole game of "memory" (§6) is about deciding what goes into the context on each turn.

3.5 The loop¶

The outer loop that calls the model, dispatches tools, and repeats. In workflow code it's usually written explicitly. In agentic code it's usually inside a framework.

3.6 Structured outputs¶

Tools depend on the model returning something your code can parse — not prose. Structured outputs are how you make that reliable. The durable idea is constrained decoding: at generation time the model is restricted to tokens that keep the output valid against a schema or grammar, so it cannot emit malformed JSON. On top of that sit two cheap habits — validate the result against a schema (JSON Schema, Zod, Pydantic), and run a bounded repair loop (feed the validation error back, ask the model to fix it, with a retry cap). Provider "JSON mode" / structured-output features implement the constrained-decoding part for you; the validation and repair are yours. This is the contract between the model and the rest of your system.

4. Stateless reality¶

4.1 The API is stateless. Period.¶

Every major LLM API — Anthropic, OpenAI, Google, all of them — is stateless. Each request must include the full conversation history you want the model to consider. The model has no memory of previous calls. None. Even if the previous call was thirty seconds ago.

This sounds obvious, but it has enormous consequences:

"Storing a conversation" is your application's responsibility. The API does nothing for you.
"Memory" is a feature you implement by choosing what to put in the context on each call.
Cost scales with input size. Longer history = more tokens = more money and more latency on every turn.
Replaying an old conversation is just sending its messages again.

A fair caveat

Some providers now expose stateful-looking abstractions — assistant/thread objects, server-side conversation state, hosted memory. These are conveniences built on the same stateless core: under the hood, the context is still assembled and resent on every call. The lesson holds — someone is always replaying the conversation; the only question is whether it's your code or the provider's.

4.2 The minimal data model for a chat application¶

A conversation is just rows. Two tables get you 80% of the way:

CREATE TABLE conversations (
    id            UUID PRIMARY KEY,
    user_id       UUID,
    title         TEXT,
    system_prompt TEXT,
    model         TEXT,
    created_at    TIMESTAMPTZ DEFAULT now(),
    updated_at    TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE messages (
    id              UUID PRIMARY KEY,
    conversation_id UUID REFERENCES conversations(id) ON DELETE CASCADE,
    role            TEXT,           -- 'user' | 'assistant' | 'tool'
    content         JSONB,          -- not TEXT — see below
    token_count     INT,
    created_at      TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX ON messages (conversation_id, created_at);

See also: PostgreSQL Backup and Restore in Docker for operating the underlying database.

4.3 Why `JSONB`, not `TEXT`¶

A "message" in modern LLM APIs is not a single string. It's a list of content blocks: text, tool-use requests, tool-results, images, documents. Flatten it to plain text and you destroy the structure that makes tool calling work. Store the message as it came off the wire.

4.4 The send-receive flow¶

1. Load history from DB, oldest first
2. Append the new user message
3. Send messages array to the model
4. Persist the user message and the assistant response

That's the entire pattern. Everything else — token budgeting, summarisation, memory retrieval — is refinement on top.

4.5 The context window problem¶

Modern context windows are large (hundreds of thousands of tokens), but not infinite. Long-running conversations eventually overflow. Two strategies:

Sliding window — drop the oldest messages once you hit a token budget. Simple, but loses information.
Compaction — periodically summarise old turns into a single "memory" message that replaces them. Preserves the gist but loses exact wording.

Production systems use compaction with a "system" message at the start that holds the running summary, plus the last N raw turns for fidelity.

5. Activation¶

5.1 Agents don't run continuously¶

A common mental model failure: imagining an agent as something always running and "thinking." Agents almost never are. They're activated by events, do work, and terminate. The interesting question is how they're activated.

5.2 The six common activation patterns¶

1. User-initiated (chatbot pattern)

A human types a message. The agent runs its loop and returns an answer. The UX most people associate with AI assistants. Best for interactive tasks where the user wants to steer.

2. Scheduled / cron

The agent runs on a fixed schedule. "Every weekday at 8am, scan my inbox and produce a digest." Good for periodic summarisation or monitoring tasks where freshness matters but not real-time.

3. Webhook / event-driven

An external event triggers the agent: a new order arrives, a customer emails support, a CI build fails. The agent activates, does its classification or processing, and exits. This is where most of the actual production agent value lives — quiet, async, no UI.

4. Email or chat-ingest

A dedicated mailbox or chat channel. Anything posted there activates an agent. Useful because users don't need to learn a new UI; they just forward an email or @-mention the bot.

5. Polling

The agent wakes up every N minutes, checks a queue or database, acts if there's work. Crude but reliable. Often used as a stopgap before proper webhooks are wired up.

6. Agent-to-agent

One agent calls another. Becomes relevant only when you have multiple specialised agents in production.

5.3 What activation looks like architecturally¶

The pattern is the same regardless of trigger:

trigger event 
  → enqueue a job (input + context references) 
  → worker picks it up 
  → agent loop runs until done 
  → result written + notification sent

The job queue is the load-bearing component. Agents are long-running, async, and failure-prone. Running them inside an HTTP request is a mistake — you'll hit timeouts, lose state on crashes, and have no retry story. Use a real job queue: Hangfire, Celery, Sidekiq, Azure Service Bus, or a workflow engine like Temporal, Inngest, or Restate.

A queue separates the trigger from the work, gives you retries, gives you visibility, gives you backpressure. Without one, an agent is a demo, not a system.

6. Memory¶

6.1 "Memory" isn't one thing¶

The single biggest mistake in agent design is treating memory as a monolithic feature. There are at least four distinct kinds, and they need different mechanisms.

6.2 The four kinds of memory¶

(a) Working memory — the running context

The current message history within a session. Lives in the API call as the messages array. Bounded by the context window. Cleared at session end. This is the "scratch pad."

(b) Episodic memory — past sessions

Summaries or transcripts of past interactions. "Yesterday we discussed the Q3 launch plan." Retrieved by similarity or recency when relevant. Used so the agent doesn't start from zero every conversation.

(c) Semantic memory — facts about the world or user

Structured statements: "The user lives in Belgium." "The product launches in Q1." Written explicitly, queried explicitly, editable. This is what ChatGPT-style "memory" features expose.

(d) Procedural memory — learned skills

Workflows, playbooks, or improved system prompts derived from past performance. The agent (or an offline process) notices it does X better when it does Y first, and updates its own approach. The most research-frontier of the four; emerging products in this space include Anthropic's Skills primitive.

6.3 The standard memory implementation¶

In practice, three components work together:

A vector database holds embeddings of past content (conversation chunks, facts, documents) for similarity search. Cheap, fuzzy, good for "remind me what we discussed about X."
A structured store (a normal SQL database) holds canonical facts: user profile, preferences, system state. Queried by ID/key.
A summariser runs after each session and condenses raw history into a short, durable record. Without this, the vector store fills with noise; with it, you get crisp episodic memory.

6.4 The retrieval pipeline¶

The memory pipeline runs at the start of each turn:

user message arrives
  ↓
embed the message
  ↓
retrieve top-K relevant memories (vector search) 
  + load canonical facts (SQL lookup)
  ↓
build system prompt: 
   base instructions + relevant memories + facts
  ↓
agent loop runs (tools, reasoning, etc.)
  ↓
on session end: 
   summarise + decide what to persist

6.5 The decay and dedupe step¶

Memory grows. Stale or contradictory facts pollute retrieval. Real production systems run a background job that:

Detects contradictions ("user lives in Ghent" vs "user moved to Brussels last month") and resolves them.
Decays old, low-utility memories.
Merges duplicates.
Lets the user audit and edit their own memory.

This last point isn't just nice-to-have. Under privacy regulation (notably GDPR), the right to rectification and erasure apply directly to derived facts about a user.

6.6 Agent-controlled memory¶

A useful pattern: expose memory operations as tools to the agent itself. remember_fact(text), update_preference(key, value), forget(id). The agent decides what's worth persisting and writes to memory as part of its normal tool-use flow. This is how Claude's and ChatGPT's user-memory features work under the hood.

7. Context engineering¶

7.1 Prompt engineering grew up¶

"Prompt engineering" was about wording a single instruction well. Context engineering is the broader discipline it became: deliberately constructing everything the model sees on a given call — system instructions, tool definitions, retrieved documents, prior turns, tool results, and memory — under a fixed token budget. Many production failures that look like "the model isn't smart enough" are really context-construction failures.

7.2 The instruction hierarchy¶

Not all context is equal. Instructions arrive in layers — system, developer, user, tool output — and the model weights them roughly in that order of authority. Two practical consequences: keep durable rules in the system layer rather than buried in a user turn, and never let untrusted tool or document text be read as instructions (see prompt injection, §10). Order matters too: models attend most to the start and end of a long context and can lose material in the middle, so put the load-bearing instructions where they'll actually be seen.

7.3 Curation beats accumulation¶

The instinct to "give the model everything" backfires: longer context is slower, costlier, and measurably worse at retrieval once the window gets crowded. The job is selection, not accumulation.

Rank and filter — don't dump every retrieval hit into the prompt. Rank by relevance and keep the top few; a reranking step usually pays for itself.
Shape tool results — a tool that returns 50 KB of JSON should be reduced to the handful of fields the model needs before the result re-enters the context.
Compress history — distil old turns into a running summary (the compaction pattern from §4) instead of carrying every raw message forward.
Budget tokens — treat the window as a fixed budget split across instructions, tools, retrieval, and history, and decide that split on purpose.

The mental model

Memory (§6) decides what could be recalled; context engineering decides what actually enters the window this turn. A great retrieval system still fails if the harness assembles the context badly.

8. Observability¶

8.1 You cannot debug what you cannot see¶

The single most common production failure mode for agents is silent failure: the agent confidently does the wrong thing, and nobody knows until a downstream system complains. Observability is not optional; it's the only way to keep agents trustworthy.

8.2 Tracing: the foundation¶

Every agent run should produce a trace — a tree of spans capturing everything the agent did. The leading vendor-neutral standard is OpenTelemetry with its GenAI semantic conventions — CNCF-backed and supported across the major observability platforms — so agent telemetry uses the same plumbing as regular distributed tracing, and the same dashboards work for it. (The GenAI-specific conventions are still stabilising, so expect some attribute churn; the core tracing model is solid.)

A useful trace contains, for each run:

The initial trigger and input
Every LLM call: model, prompt, response, token usage, latency, cost
Every tool call: name, arguments, result, duration, success/failure
Intermediate reasoning, if the model exposes thinking blocks
Decision branches: which route taken, which sub-agent invoked
Final output
Any human interventions

In the dashboard this renders as a waterfall — exactly like an HTTP distributed trace, but with each LLM call expandable to show prompt and response.

8.3 Tooling landscape¶

Many products implement the same OpenTelemetry conventions:

LangSmith — most mature LLM-tracing UI; works beyond LangChain.
Langfuse — open-source, self-hostable.
Helicone — proxy-based, minimal setup.
Arize Phoenix — open-source, strong on evaluation alongside tracing.
Weights & Biases Weave — for teams already using W&B.
Datadog, New Relic, Honeycomb — general-purpose APMs with LLM features.
.NET Aspire dashboard — for .NET developers; built-in OTel waterfall.

The choice matters less than the discipline of having one.

8.4 Evals: testing the untestable¶

Traditional unit tests don't work well for LLM-driven systems because outputs are stochastic. Evals are graded assertions on LLM output: "Did the agent extract the correct invoice total?" "Is the response factually consistent with the source documents?" "Is the tone appropriate?"

Evals run on a regression set after every prompt or model change. They catch silent quality drops that traditional tests miss. Mature teams treat eval coverage with the same seriousness as test coverage. It helps to know the main kinds:

Offline vs online — graded against a fixed dataset before deploy, versus sampled from live production traffic.
Component vs trajectory vs end-task — score one step (a retrieval, a single tool call), the whole reasoning path the agent took, or simply whether the final task was completed.
LLM-as-judge and human-preference — for open-ended output with no exact answer, a second model or a human rates quality.
Safety evals — jailbreaks, harmful content, and PII leakage, run as their own suite.

Common eval tools: LangSmith, Langfuse, Braintrust, Promptfoo, Phoenix.

8.5 Human-in-the-loop checkpoints¶

Some actions are too consequential to let the agent execute autonomously. HITL checkpoints are explicit pause points where the agent waits for human approval before proceeding: sending an email, posting a financial transaction, modifying production infrastructure.

The trace UI is usually where the human reviews context and approves or rejects. HITL is the primary control mechanism for bounded autonomy — giving the agent freedom within a safe envelope.

9. Code execution¶

9.1 Why this section exists¶

This is the topic that confuses people most because it looks like magic from the outside. "How does the assistant run code?" Once you understand the mechanism, every other agent capability becomes easier to reason about.

9.2 The mechanism¶

A code-executing agent is built from three pieces:

A language model that, on each turn, sees the conversation plus a list of available tools with their schemas.
A sandboxed environment sitting next to the model — a fresh Linux container with Python, Node, bash, a filesystem, and limited network access.
A tool harness that:
exposes the sandbox's capabilities to the model as tool schemas (run_bash, write_file, read_file, etc.),
intercepts every tool call the model emits,
executes it in the sandbox,
feeds the result back to the model as the next message,
loops until the model stops emitting tool calls.

That's it. The model never executes anything directly. It emits intent (a structured tool call); a separate system executes; results flow back as text. The same separation we established in 3.3 The harness applies here, with the tools being shell and file operations.

9.3 What "I'll take a screenshot" really is¶

When an assistant says "I'll render this HTML and take a screenshot," the mechanism is identical:

Model emits: take_screenshot(html_path="/tmp/page.html", out="/tmp/shot.png")
  → harness launches headless Chromium in the sandbox
  → captures the PNG
  → writes it to disk
  → returns the path to the model
  → the client displays the PNG inline

No special capability. Just one more tool.

9.4 The variants¶

The architectural pattern is the same; only the breadth of the sandbox differs:

Code interpreter — sandbox is Python-only. Narrow, safest.
Bash sandbox — sandbox is a full Linux container. Broad, more useful.
Browser use — sandbox is a browser driven via DOM operations. Used for web scraping, form-filling, QA testing.
Computer use — sandbox is a virtual desktop driven via screenshots, mouse clicks, and keystrokes. The most general, the most expensive, the most fragile.

Each is "agent with a tool" — what changes is what the tool can do.

10. Sandbox architectures¶

10.1 Three layers, not two¶

When people first build a sandboxed agent, they tend to collapse everything into two layers — "the agent" and "the sandbox." A clearer mental model has three:

┌────────────────────────────────────────────┐
│  1. Agent process                          │
│     - the loop                             │
│     - the LLM client                       │
│     - the harness (tool dispatch logic)    │
│     - tool implementations                 │
└────────────────────┬───────────────────────┘
                     │  HTTP / SDK
                     ▼
┌────────────────────────────────────────────┐
│  2. Sandbox control plane                  │
│     - starts/stops containers              │
│     - exposes exec / file / net endpoints  │
│     - enforces quotas, timeouts, isolation │
└────────────────────┬───────────────────────┘
                     │
                     ▼
┌────────────────────────────────────────────┐
│  3. Sandbox runtime                        │
│     - the container itself                 │
│     - executes the commands                │
│     - ephemeral filesystem                 │
└────────────────────────────────────────────┘

The harness lives in layer 1. The "API in front" of the sandbox is layer 2. The container is layer 3.

10.2 Where the sandbox API comes from¶

There are three realistic options for the control plane.

Option A — The Docker Engine API directly

The Docker daemon already exposes an HTTP API on a local socket. The agent talks to it directly via a library (e.g., Docker.DotNet in C#, docker-py in Python, dockerode in Node).

Pros: zero extra infrastructure, fast, simple. Cons: the agent process needs Docker socket access, which is effectively root on the host. Fine for local development, dicey for shared production hosts.

Option B — A custom sandbox service

You build a thin HTTP service in front of Docker. The agent talks to your service; your service talks to Docker.

agent ──HTTP──▶ sandbox-svc ──Docker API──▶ container

The service exposes endpoints like POST /sessions, POST /sessions/{id}/exec, POST /sessions/{id}/files. The agent has only an HTTP token; Docker access is contained to the service.

Pros: smaller blast radius if the agent is compromised, centralised quotas and audit logs, swap the backend without touching agents. Cons: one more service to operate.

Option C — Managed sandbox-as-a-service

Skip building the control plane entirely. Use a vendor: E2B, Modal, Daytona, Runloop. They give you an SDK that spins up sandboxes via API.

Pros: zero ops, fast to ship. Cons: data sovereignty constraints, vendor lock-in, recurring cost.

10.3 Which to choose¶

A practical progression:

Prototype: option C. Get to working code in days.
Internal tools: option A. The Docker socket exposure is acceptable on a single-tenant dev box.
Production with sensitive data: option B. Worth the operational cost to confine Docker privileges.

10.4 What goes inside the sandbox container¶

A typical sandbox image:

A small base OS (Alpine or Debian-slim).
The runtimes your agent needs: Python, Node, perhaps a headless browser.
A non-root user account that the agent's tools run as.
A working directory mounted as tmpfs or an ephemeral volume.
No host filesystem mounts. No SSH keys. No secrets baked in.

Resource limits are set at container creation: memory cap, CPU cap, process count cap, wall-clock timeout. Ephemeral containers are the default — destroy at session end, recreate next time.

10.5 The hard parts of sandboxing¶

The above is the easy part. Production-grade sandboxing requires:

Network egress allow-listing. The sandbox should not be able to reach the open internet unless you explicitly permit a target. Otherwise a prompt-injected agent will gleefully exfiltrate data.
Secret isolation. Secrets the agent needs (API keys, DB credentials) should be injected at the tool layer, not the sandbox layer. The model and the sandbox never see them; the tool signs requests on the agent's behalf.
Resource quotas. A bad infinite loop shouldn't cost €500.
Audit logging. Every command, file write, and network call gets logged with the conversation ID. This is the forensic trail.
Filesystem hygiene. Even ephemeral containers can leak data if built carelessly. Don't bake credentials into base images.

10.6 The recurring security threat: prompt injection¶

A code-executing agent is, by definition, executing arbitrary code that a language model decided to write. If the model is jailbroken or prompt-injected via untrusted input (an email it reads, a webpage it scrapes, a document it summarises), it may happily run rm -rf / or copy your secrets to an attacker.

The mitigation is not "trust the model." It's the sandbox itself plus the egress controls plus the secret isolation. Defence in depth. Sandboxing isn't optional for any agent that touches untrusted input.

11. MCP¶

11.1 The integration problem¶

A typical organisation runs many systems: CRM, ERP, ticketing, document store, codebase, calendar, email. An agent that's useful in this environment needs to talk to several of them.

The naive approach is to build custom integrations: one tool per business system, all bespoke for the specific agent framework you're using. This explodes combinatorially — N agent frameworks times M business systems equals NxM integrations.

11.2 What MCP is¶

The Model Context Protocol (MCP) is a standardised protocol for connecting AI systems to data and tools. Originally developed by Anthropic and now adopted across the major model providers (Anthropic, OpenAI, Google, Microsoft, AWS). In late 2025 it was donated to the Linux Foundation's Agentic AI Foundation, removing single-vendor risk and cementing it as a neutral, durable standard. Adoption is broad — though implementation depth (auth, governance, transport) still varies by platform.

The analogy that sticks: MCP is the USB-C of AI integrations. One protocol, many clients, many servers.

11.3 The model¶

MCP defines three primitives a server can expose:

Tools — executable functions the model can call.
Resources — read-only data the model can fetch (documents, database rows, API responses).
Prompts — reusable prompt templates.

An MCP server wraps a system of record and exposes its capabilities through these primitives. An MCP client (any AI assistant — Claude Desktop, ChatGPT, an in-house agent) can connect to any MCP server and use its tools.

11.4 Why this matters architecturally¶

If you build one MCP server over a business system, every AI assistant your organisation deploys can use it. The investment compounds. The old "build custom integration for Framework X" becomes "build the MCP server once; use it from all frameworks and assistants."

For a developer building agents, this also means: before building a custom tool, check whether an MCP server already exists. The ecosystem is growing fast. Many SaaS products now ship official MCP servers.

11.5 MCP servers as architecture, not just integration¶

A subtle but important point: exposing your own agent as an MCP server turns it into a reusable building block. A specialised internal agent can be consumed not just by your own app but by Claude Desktop, ChatGPT, Copilot, or another agent. One investment, many surfaces.

This is the architecture that scales across an organisation: each specialised capability is an MCP server; assistants are clients that compose them.

12. Build options¶

The framework landscape changes quickly. The categories are stable; the specific tools shift every six months. Use this section as a map rather than a recommendation.

12.1 Code-first frameworks¶

You write Python, TypeScript, or C#. You control the loop, debug your own prompts, own the dependencies. Maximum flexibility, maximum responsibility.

Python is the centre of gravity. Notable frameworks:

LangChain / LangGraph — most widely used. LangChain is the toolkit; LangGraph is its newer state-machine layer for proper agent loops with checkpointing.
LlamaIndex — RAG-first, strong on document/data workflows.
CrewAI — opinionated multi-agent framework.
AutoGen — Microsoft Research's multi-agent conversation framework.
Pydantic AI — lean, type-safe, FastAPI-style.
Smolagents — minimalist, ~1000 lines of code.

Provider-native SDKs are increasingly capable on their own and often the right choice over a heavy framework:

Anthropic Claude SDK — direct tool use, computer use, native MCP support, official .NET/Python/TypeScript packages.
OpenAI Agents SDK — native tool calling, traces, MCP support.
Google ADK — Gemini-native, tight Google Cloud integration.

.NET ecosystem:

Microsoft Agent Framework — consolidated successor to Semantic Kernel and AutoGen for .NET. Sits on Microsoft.Extensions.AI, provider-agnostic, MCP-native.
Semantic Kernel — predecessor; in maintenance, with a compatibility bridge to Agent Framework.

Java ecosystem:

Spring AI — equivalent of Semantic Kernel for the Spring world.

12.2 Low-code / visual orchestrators¶

Drag-and-drop nodes, but you still write some prompts and logic. The "iPaaS-for-AI" tier.

n8n — open-source, self-hostable, strong AI agent node.
Flowise, Langflow — visual LangChain.
Dify — open-source LLM app platform with built-in RAG and chat UI.
Make.com, Zapier — classic iPaaS with AI nodes added.

12.3 No-code agent builders¶

Point-and-click, often vertical (sales, support, RPA).

Microsoft Copilot Studio — for agents inside Microsoft 365.
Salesforce Agentforce — for Salesforce environments.
ServiceNow AI Agents — for ServiceNow.
Lindy, Relevance AI, Stack AI — SMB-focused.

12.4 Self-hosted open-source¶

For data-sovereignty constraints.

Ollama, LM Studio, vLLM — run open-weight models locally.
OpenWebUI + Ollama + n8n — a popular fully-local stack.
Mistral / Le Chat — French-hosted, GDPR-friendly.

12.5 The decision¶

Start with the smallest abstraction that gets the job done. Provider-native SDKs are often enough. Reach for frameworks when you need their specific affordances (multi-agent orchestration, complex state machines, evals integration). Reach for no-code only when the agent lives entirely inside an existing platform.

The framework choice is secondary to the architectural choices in the preceding sections.

13. Cost architecture¶

13.1 Why cost is structural¶

Agents are token-hungry by nature: a single request can fan out into many LLM calls, each carrying a context that grows every turn (§4). A workflow step costing a fraction of a cent becomes an agent run costing euros. The levers below are durable — they outlive any specific model or price list — and most are architectural, not parameters you tune at the end.

13.2 Spend less per call¶

Prompt caching — providers can cache the unchanging head of a prompt (system instructions, tool definitions, long context) so repeated calls pay a fraction of the cost and latency. Decisive for agents that resend a large context on every loop.
Semantic caching — key responses by embedding similarity so near-duplicate queries return a stored answer with no model call at all.
Batch inference — for work that isn't time-critical, submit requests as an offline batch, typically at a steep discount.
Trim the context — context engineering (§7) is also a cost lever: every token you don't send is money you don't spend, on every turn.

13.3 Use a cheaper model when you can¶

Model routing sends easy requests to a small, cheap model and only the hard ones to a frontier model — a classifier or heuristic decides. The common shape is small-model-first with escalation: try the cheap model, escalate only when confidence is low or the task is flagged hard. Most traffic is easy, so routing captures that saving without giving up quality on the hard tail.

Incoming request
        │
        ▼
Classify difficulty
        │
        ▼
Cheap model handles the easy majority
        │
        ▼
Escalate the hard tail to a frontier model

The discipline

Instrument cost per run from day one — it rides along in your traces (§8). Set budgets, and treat a runaway agent the way you would treat a memory leak. Bounded autonomy is itself a cost control: fewer wasted loops, less spend.

14. Pitfalls¶

14.1 Treating an agent as a chatbot with more buttons¶

The single most expensive mistake. Even when the agent is user-initiated (§5.2), it is not a synchronous request-response system. A single user turn can fan out into many LLM calls and many tool invocations over seconds or minutes. Build agents as background work driven by triggers rather than as request handlers: queues, traces, retries, idempotency keys. The user-facing chat interface is just one trigger type sitting in front of that work, not a substitute for it.

14.2 Skipping the workflow stage¶

The lure of "give the agent goals, let it figure it out" is strong. The reality is that hand-orchestrated workflows are more reliable, more debuggable, cheaper, and faster for the overwhelming majority of useful tasks. Reserve full autonomy for cases where you genuinely cannot pre-script the path.

14.3 Hallucination treated as a debug task¶

LLM hallucinations are not bugs to fix; they are a property of the underlying technology. The mitigation is structural, not corrective:

Ground responses in retrieved facts (RAG).
Make the agent cite sources.
Validate critical outputs with a second LLM, a deterministic checker, or a human (HITL).
Limit blast radius via bounded autonomy.

14.4 No observability until something breaks¶

Adding tracing after the fact is much harder than adding it from day one. The instrumentation is a one-time cost. The benefit accrues forever. Bake in OpenTelemetry from the first commit.

14.5 Conflating the four kinds of memory¶

Designing "memory" as one feature leads to systems where the agent remembers the wrong things, forgets the right things, and contradicts itself. Build for the four kinds explicitly: working / episodic / semantic / procedural.

14.6 Cost surprises¶

A chat that costs €0.01 per turn becomes a tool-using agent that costs €0.50 per turn — twenty LLM calls, longer prompts, retries on failure. This is not a bug; it's the cost of intelligence. Plan for it, instrument it, apply bounded autonomy where the cost is disproportionate to value, and treat cost as an architectural concern in its own right (see §13).

14.7 Underestimating prompt injection¶

If an agent reads any input not authored by your developers — emails, web pages, documents, API responses, user-uploaded files — that input can carry instructions to the model. Treat all such input as adversarial. Defence: sandboxing, egress controls, output validation, HITL on sensitive actions.

14.8 Tool descriptions written like comments¶

The model decides whether and how to call a tool based on its description and schema. Vague descriptions produce unreliable tool use. Treat tool docs as user-facing documentation for an LLM "user." Be precise about what the tool does, when to use it, and what its arguments mean.

15. Reference architecture¶

Pulling it together, a production-grade agent has these components, each fulfilling a clear role:

┌─────────────────────────────────────────────────────────────┐
│  Triggers                                                   │
│  webhook | scheduler | chat UI | email | queue              │
└──────────────────────────┬──────────────────────────────────┘
                           ▼
┌─────────────────────────────────────────────────────────────┐
│  Job Queue  (e.g., Hangfire / Celery / Temporal)            │
│  - durability, retries, backpressure                        │
└──────────────────────────┬──────────────────────────────────┘
                           ▼
┌─────────────────────────────────────────────────────────────┐
│  Agent Worker                                               │
│  ┌──────────────────────────────────────────────────────┐   │
│  │  Harness  (framework or hand-rolled)                 │   │
│  │   - schema generation                                │   │
│  │   - the loop                                         │   │
│  │   - tool dispatch                                    │   │
│  └──────┬───────────────────────────────────────────────┘   │
│         │                                                   │
│  ┌──────▼──────┐  ┌──────────────┐  ┌─────────────────┐     │
│  │  LLM Client │  │ Memory Layer │  │ Tools           │     │
│  │  (OpenAI/   │  │ - working    │  │ - business APIs │     │
│  │   Anthropic/│  │ - episodic   │  │ - MCP servers   │     │
│  │   Ollama)   │  │ - semantic   │  │ - sandbox exec  │     │
│  └─────────────┘  └──────┬───────┘  └────────┬────────┘     │
└─────────────────────────┼─────────────────────┼─────────────┘
                          ▼                     ▼
              ┌──────────────────┐   ┌─────────────────────┐
              │  Vector DB + SQL │   │  Sandbox / MCP /    │
              │  - history       │   │  Internal Systems   │
              │  - facts         │   │  - via control plane│
              │  - summaries     │   └─────────────────────┘
              └──────────────────┘

                          OpenTelemetry traces flow
                          from every component to:

              ┌─────────────────────────────────┐
              │  Observability platform         │
              │  (Langfuse, LangSmith, Aspire,  │
              │   Datadog, etc.)                │
              └─────────────────────────────────┘

The seams in this diagram match the conceptual sections of this guide:

Triggers + queue → §5 (activation).
Harness + LLM client → §3 (anatomy), §4 (stateless reality).
Memory layer → §6.
Tools → §3.2, including sandbox tools (§9, §10) and MCP-backed tools (§11).
Observability → §8.

Every architectural decision is a choice within one of these seams. Whenever a design question arises, the first step is to locate it in this picture.

16. Glossary¶

Agent — a system that uses a language model as a reasoning engine to take actions via tools.
Augmented LLM — an LLM enhanced with tools, memory, and retrieval; the basic unit on which agents are built.
Bounded autonomy — the design principle of giving an agent only as much freedom as the task requires, no more.
Compaction — replacing old conversation turns with a summary to keep context windows manageable.
Constrained decoding — restricting generation to tokens that keep the output valid against a schema or grammar, guaranteeing well-formed structured output.
Context engineering — the discipline of deliberately selecting, ordering, and compressing everything placed in the model's context window on each call.
Eval — a graded assertion on LLM output used as a regression test for LLM-driven systems.
HITL — human-in-the-loop; an explicit pause for human approval.
Harness — the code between the model and the tools, dispatching tool calls and feeding back results.
MCP — Model Context Protocol; standardised protocol for connecting AI assistants to tools and data.
Model routing — directing each request to the cheapest model that can handle it, escalating only the hard cases to a frontier model.
Prompt caching — reusing the unchanging prefix of a prompt across calls so repeated requests cost a fraction of the tokens and latency.
RAG — retrieval-augmented generation; injecting retrieved information into the prompt.
ReAct — the canonical reasoning-and-acting loop pattern.
Sandbox — an isolated environment where the agent's tools execute arbitrary commands.
Semantic caching — caching responses keyed by meaning (embedding similarity) so near-duplicate queries skip the model entirely.
Span / trace — units of OpenTelemetry observability; spans represent operations, traces are trees of spans.
Stateless API — an API that retains no state between requests; the entire conversation must be replayed each call.
Tool — an external function the model can call, defined by a name, description, schema, and implementation.
Workflow — a hand-orchestrated pipeline of LLM calls, as opposed to an autonomous agent that chooses its own path.
Working / episodic / semantic / procedural memory — the four distinct kinds of agent memory, each with different mechanisms.

Concepts referenced as wiki-links in this guide are placeholders for notes that may not yet exist in this vault. Hover for a list, or click any to create the corresponding stub note:

ReAct
RAG — covered in practice by agent-memory-at-scale (RAG as an agent search tool rather than a prompt-stuffing stage)
JSON Schema
LangChain
OpenTelemetry
vector database — covered in practice by agent-memory-at-scale (when one is actually warranted over files)
Model Context Protocol
prompt injection
Microsoft Agent Framework
PostgreSQL Backup and Restore in Docker (existing)
claude-code-memory-architecture (existing) — a production case study of §6.6 agent-controlled memory and §7 context engineering
agent-memory-at-scale (existing) — file-and-index memory vs. the §6.3–6.4 vector pipeline; note the tension: that page argues the pre-stuffed top-K pipeline described in §6.4 is the legacy shape, superseded by agent-initiated search
ai-adoption-maturity-model (existing) — the organisational ladder above these mechanisms (bottleneck per step, guardrail escalation); note the tension with the multi-agent scepticism in §2.3