Loading…
AI Agent Developer: San Francisco
San Francisco SaaS companies and AI teams know the terminology. What they run into is the engineering gap between a working demo and a reliable production system. We close that gap.
The failure modes are predictable: reliability that compounds across tool calls, evaluation that only checks outputs instead of decision trajectories, memory architecture that conflates three separate concerns, and tool schemas written too loosely for the model to call correctly. We fix these specifically.
Every project is fixed scope. Discovery call first, then a written scope before any work starts.
Describe where your agent is breaking down.
Reliability compounds across tool calls. If each individual tool call succeeds 90% of the time and your agent makes five tool calls per task, the probability that all five succeed is 0.9^5, which is 59%. A five-step agent that feels reliable in testing is failing 41% of the time in production. The fix is not better prompting. It is retry logic, structured output validation at each step, and fallback chains.
Planning agents that use ReAct or Tree-of-Thought patterns require trajectory evaluation, not output evaluation. Checking whether the final answer is correct tells you nothing about whether the agent took the correct path to get there. Trajectory evals record every tool call, every intermediate reasoning step, and every decision branch, then score the sequence against a reference.
Memory architecture for production agents has three distinct layers. Short-term working memory is the task's current context window. Long-term episodic memory is a vector store of past interactions and documents, retrieved by semantic similarity. Procedural memory records which tools are reliable for which task types. These are separate engineering problems with separate data stores and retrieval strategies.
Specific problems, each with a specific engineering answer.
Structured outputs at every tool call step. Retry logic with exponential backoff. Fallback chains that route to alternative strategies when primary tools fail. Each step is validated before the next begins.
We build trajectory eval harnesses using LangSmith or a custom evaluation pipeline. Test suites cover task completion rate, tool-call accuracy, hallucination rate, and decision path correctness, not just final output quality.
Three separate data stores for three separate concerns. Short-term context in the model's context window. Episodic memory in a Pinecone or pgvector store. Procedural memory in a structured database with explicit lookup logic.
Tool calling reliability depends more on schema quality than model capability. We write tight JSON schemas with explicit parameter descriptions, enumerated valid values, and example calls. The model calls tools correctly when the schema is precise.
We start with the failure budget. Before writing production code, we map every tool call in the agent's workflow and assign a reliability target. If a step cannot meet its target with the primary tool, we spec the fallback before building.
Every tool call is instrumented from day one: input, output, latency, and cost go into a log store queryable for debugging and compliance. LangSmith works well for this. We can also build a custom logging layer if you have specific retention or audit requirements.
For SaaS products shipping agents as a feature, we enforce tenant isolation at the tool layer. Each tool call carries a tenant context that the authorization layer validates. The LLM cannot route around this. It is not a system prompt instruction, it is a hard constraint in the tool execution layer.
Evaluation runs on every commit via CI. We use a fixed set of scenario inputs and score trajectory correctness, output accuracy, and latency. Regressions block the deploy.
Tell us what your agent does today and where it breaks. We'll reply within one business day with a technical assessment and rough scope.