Most AI demos look simple: type a message, get an answer. Production AI agents are not that. They run loops, call external systems, manage state across turns, and make decisions about what to do next. Understanding the three core architectural patterns is the difference between an agent that works in a Colab notebook and one that handles 10,000 requests a day without breaking.
This post covers the agent loop, tool calling architecture, and memory systems — the three building blocks of every production agent we've shipped.
The agent loop: perceive, decide, act, observe
Every AI agent runs some version of this loop:
- Perceive — receive input (user message, tool result, system event)
- Decide — send the current context to the LLM and get a response
- Act — if the LLM requested a tool call, execute it; if not, return the final answer
- Observe — inject the tool result back into context and loop again
The loop continues until the LLM returns a final answer without requesting further tool calls, or until a maximum step count is hit (a hard limit you should always set — runaway agents are a real operational risk).
In code, the loop looks roughly like this:
while steps < MAX_STEPS:
response = llm.complete(messages)
if response.stop_reason == "end_turn":
return response.content
tool_calls = response.tool_calls
results = execute_tools(tool_calls)
messages.extend([response, *results])
steps += 1
raise MaxStepsExceeded()
Simple loops like this handle the majority of single-agent tasks. Multi-agent architectures add an orchestrator layer that routes between sub-agents, but the individual agents still run this same pattern internally.
Why the loop matters operationally
The loop design determines your latency profile and your cost exposure. Each iteration is a round-trip LLM call — at $0.003/1K tokens, a 10-step agent on a 10K-token context costs $0.30 per run. At 1,000 runs/day, that's $3,000/month from a single agent.
This is why tool definitions and prompt design matter before you ever talk about memory systems. A well-designed agent completes most tasks in 2–4 steps. A poorly designed one routinely hits 10–15 steps on the same tasks.
Tool calling architecture
Tool calling gives the LLM a structured way to request external actions — database queries, API calls, calculations, file reads. The LLM doesn't execute these directly; it produces a structured output saying "call this function with these arguments," and your application code handles the actual execution.
Tool definition design
Every tool needs three things to work reliably:
- A clear, unambiguous name.
search_knowledge_baseis better thansearch.get_customer_order_historyis better thanget_orders. - A specific description. The description is part of what the LLM reads to decide whether to use the tool. Vague descriptions produce wrong tool choices. Include: what the tool does, when to use it, what it returns.
- Typed, constrained parameters. Use enums where possible. Add descriptions to every parameter. The LLM fills these in from context — the more constrained your schema, the fewer hallucinated parameter values you'll see.
Parallel vs sequential tool calls
Modern LLMs support requesting multiple tool calls in a single turn. This is a significant performance gain: instead of running three database queries in three round trips (3× LLM latency), you run them in parallel in one round trip.
Use parallel tool calls when the tool results are independent of each other. Use sequential calls when one tool's output is the input to the next. Most agents should be designed to maximize parallel execution — it cuts both latency and cost.
Handling tool errors
Tools fail. APIs time out. The LLM sometimes passes invalid arguments. Your loop needs to handle tool errors gracefully and inject the error back into context so the LLM can decide what to do next (retry, try a different tool, or ask the user for clarification).
Do not let a tool error silently kill the loop. Surface it to the LLM. Nine times out of ten, a well-prompted agent will recover on its own.
Memory types and when to use each
"Memory" in agent systems is overloaded. There are four distinct types, and conflating them leads to architecture mistakes.
| Memory type | Where it lives | Typical size | Use case | |---|---|---|---| | Working memory | Context window | Up to model limit | Current conversation, recent tool results | | Semantic memory | Vector database | Millions of chunks | Knowledge base retrieval (RAG) | | Episodic memory | Database | Unbounded | Conversation history, past interactions | | Procedural memory | Prompts or fine-tune | Small | Cached reasoning, personality, task patterns |
Working memory
This is the context window — everything the model can "see" in a single call. Modern context windows are large (128K tokens is common; some models go to 1M+), but you're paying for every token, so filling them carelessly is expensive.
For most agents, working memory should hold: the system prompt, the current conversation, and the most recent tool results. Summarize and evict older content as conversations grow.
Semantic memory (the RAG layer)
When an agent needs to answer questions from a large knowledge base, you don't put the whole knowledge base in the context window. You retrieve the relevant chunks at query time and inject only those. This is the RAG pattern — and it's worth its own deep dive, but architecturally, it's just a retrieval tool the agent can call.
The common mistake: treating semantic memory as a substitute for working memory. They serve different purposes. RAG retrieves relevant knowledge; working memory holds the current reasoning context.
Episodic memory
Long-running agents need to remember what happened in past sessions. A customer support agent should know that this user called last week about a billing issue. A sales development agent should remember that this prospect asked about pricing in a previous conversation.
The standard pattern: summarize each conversation at the end and write the summary to a database keyed by user ID. At the start of a new session, retrieve the N most relevant past summaries and inject them into the system prompt. The summary step is important — storing raw transcripts is expensive and noisy.
Procedural memory
This is the most underused type. Certain agents benefit from cached reasoning patterns — sequences of steps that are known to work for specific task types. These live in the system prompt as few-shot examples or chain-of-thought templates. You don't retrieve them dynamically; they're always present.
Procedural memory is particularly valuable when you need consistent behavior across a narrow, well-defined task domain.
The most common architecture mistakes
1. No step limit on the loop. Without a hard cap, a confused agent will run until you cut it off manually or your budget runs out.
2. Treating the context window as a database. Appending every tool result to the context indefinitely produces bloated, expensive, degraded calls. Summarize and evict.
3. Tool definitions written for humans, not models. The LLM reads tool descriptions to decide what to call. Write them for the model, not for your own documentation.
4. Skipping episodic memory because it's complex. Users expect agents to remember context across sessions. Stateless agents feel broken to end users even when they're technically working.
5. One massive agent instead of specialized sub-agents. If your agent has 20 tools, consider whether some of those tools belong to a sub-agent that gets called when needed. Specialization improves reliability.
Starting architecture for most production agents
For most AI agent development projects, start here:
- Agent loop with a 15-step hard cap
- 4–8 well-scoped tools with explicit descriptions
- Working memory limited to the last 10–15 turns (summarize beyond that)
- RAG layer if there's a knowledge base larger than a few thousand tokens
- Episodic memory if users return to the same agent across sessions
Add complexity when you have evidence you need it. The most reliable production agents are simpler than their builders expected.