Most companies that get burned by an AI agent project didn't ask the wrong questions — they asked no questions at all. They saw a demo, liked the demo, signed a contract, and six months later had a prototype that didn't work on real data.
This guide gives you a structured way to evaluate AI agent development companies before you hand over money. None of this is theory — these are the signals that separate shops with shipped production agents from shops with a good slide deck.
What "shipped" actually means
Before we get into evaluation criteria, clarify the definition of shipped with any vendor you're considering.
A shipped agent is one that:
- Runs in production, not a sandbox
- Has real users generating real inputs
- Has been running long enough to break at least once
- Has monitoring, logging, and someone responsible for it
A demo is not shipped. A proof of concept is not shipped. A "pilot with 5 internal users" is marginally shipped. Ask vendors: how many agents do you currently have running in production with external users? The answer is revealing.
Six things to evaluate
1. Portfolio of shipped agents, not just demos
Ask to see two or three previous projects where the agent is live in production. Request specifics:
- What was the agent's task?
- What happens when the agent encounters an edge case it wasn't trained on?
- What did failure look like, and how was it handled?
Vendors who have actually shipped agents will answer these questions easily. Vendors who have mostly done demos will pivot to talking about the technology stack or the demo itself.
What good looks like: "We shipped a document classification agent for a legal firm. It processes about 800 documents per day. When confidence falls below 0.7, it routes to a human reviewer. Here's the false-positive rate over the last 90 days."
What a red flag looks like: "We've built agents using GPT-5, LangChain, and RAG — let us show you what we built for a similar client." Then the demo uses hand-selected inputs.
2. Eval methodology — do they have one?
This is the single most reliable signal of a real AI engineering shop vs. a demo factory.
Evals (evaluations) are the test suites that measure whether an agent is doing its job correctly. They answer questions like:
- Does the agent return the right answer on a representative set of inputs?
- Does it fail gracefully on adversarial inputs?
- Did performance get worse when we changed the prompt?
Ask any prospective vendor: "How do you measure whether your agent is working?" A shop that has shipped production agents will talk about offline eval sets, online metrics like user correction rates, and regression testing when models change.
A shop that hasn't will talk about "testing it thoroughly" or "making sure it works before launch."
Specific things to ask:
- What eval framework do you use? (Ragas, DeepEval, ROUGE, custom harness — any reasonable answer is fine)
- How many test cases are in your eval suite for a typical project?
- What metric do you optimize for, and how do you know when it's good enough?
- Can you show me an eval report from a previous project?
If they can't produce an eval report from a previous project, they don't have eval discipline. That means you won't be able to tell if the agent is improving or degrading over time.
3. Pricing transparency
Vague pricing is a reliable predictor of scope creep and invoice surprises. A serious shop can give you:
- A fixed price for a defined scope with clear change-order triggers
- A time-and-materials rate with a ceiling
- A retainer with explicitly listed deliverables
What they should not give you: "It depends on your needs" with no anchoring numbers. Ask for a ballpark even before scoping. If they refuse to name any number until they've done a full discovery, that's fine — but the number should come at the end of discovery, not whenever you finally stop pushing.
Typical market rates as of 2025:
- Freelance AI engineer: $120–$200/hr
- Small AI agency: $15,000–$30,000/month for ongoing work
- Fixed-scope agent build: $20,000–$80,000 depending on complexity
If a vendor quotes you significantly below these ranges, ask why. It's either offshore, junior, or padded with out-of-scope expansion later.
4. Code ownership and IP
You should own the code on day one. Not at the end of the project. Not after a final payment. On day one.
This means:
- Your repo, not theirs
- Commits going directly to your version control
- No proprietary middleware or wrappers you can't replace
- The right to hand the code to a different vendor
Some shops use their own internal frameworks and bill you for access to them. That's a lock-in play. Fine if the framework is genuinely better and you understand the trade-off. Not fine if it's disguised as a dependency you can't escape.
Ask directly: "If we end the engagement today, do we leave with everything we need to run and maintain this agent without you?" If the answer is anything other than yes, push until it is, or walk.
5. Communication cadence
How a vendor communicates during sales is exactly how they'll communicate during the project.
Ask: "What does a typical week look like during an active build?" A good answer includes:
- A defined async update rhythm (daily standup in Slack, weekly written status report)
- A stated response time for critical issues
- A single point of contact who can make decisions
If they promise "constant communication" without defining it, that means ad-hoc communication which means silence when things are going wrong.
6. Post-launch ownership
This is where most AI projects fall apart. The agent ships. Six months later the underlying model changes. Prompts that worked stop working. Context lengths shift. The vendor is on the next project.
Before signing, establish:
- Who owns prompt maintenance when the LLM provider changes a model?
- What's the SLA if the agent starts returning errors in production?
- Is there a defined handoff process if you want to take over maintenance?
- What's included in post-launch support, at what cost?
An agent isn't software you deploy once and forget. It requires ongoing calibration. A vendor who pretends otherwise has either never maintained a production agent or is planning to up-sell you later.
Red flags that should stop negotiations
"We can achieve 100% accuracy." No production agent achieves 100% accuracy on real-world inputs. Anyone promising this either doesn't understand the problem or is planning to hide failures from you.
No mention of how they test. If evals don't come up naturally in the first conversation, ask directly. If the answer is "we test it manually before delivery," that's not a testing methodology.
They won't give you the code. This shouldn't need to be said, but some shops make this a negotiation point. It shouldn't be one.
They only know one stack. A shop that uses LangChain for everything, or GPT-5 for everything, regardless of the problem, isn't engineering — they're assembling kits.
References but no specifics. "We can connect you with happy clients" is fine, but if every reference gives the same generic positive review with no operational details, that's coordinated social proof, not evidence.
Questions to ask in the first call
Bring these to the initial conversation. How they answer tells you more than any proposal:
- Show me an agent you've shipped to production — what does it do, and what does failure look like?
- How do you measure whether an agent is performing correctly?
- What eval framework or tooling do you use?
- If I stop this engagement tomorrow, do I leave with everything I need to keep it running?
- Who owns the code during the engagement?
- What happens when the LLM provider changes the model you're using?
- Can I see a real eval report from a previous client project?
Why you should see an eval report before signing
Asking for an eval report from a previous project (with the client's permission, or anonymized) is the most efficient filter available. A real eval report shows:
- A defined test set with question/answer pairs or task/output pairs
- A metric (accuracy, F1, pass rate, retrieval recall)
- A threshold the project was targeting
- Results before and after optimization
If a vendor can't produce this for any previous project, they haven't been measuring whether their agents work. You would be their first client whose outcome they track systematically. That is not a position you want to be in.
Vendors who do have eval discipline will often share these reports proactively because they're proud of the numbers. That eagerness is itself a positive signal.
The bottom line
Choosing an AI agent development company comes down to one underlying question: have they shipped real agents to production and do they have the measurement infrastructure to know if those agents are working?
The six criteria above — shipped portfolio, eval methodology, pricing transparency, code ownership, communication cadence, and post-launch plan — all trace back to that question. A company that can answer all six clearly has probably shipped agents. A company that can't answer two or three of them has probably shown a lot of demos.
See a real eval report before you sign anything. If they don't have one, that's your answer.
Explore our AI agent development services or learn about how we approach agent evaluation.