Most vendor evaluations rely on proposals, references, and demos. Proposals are sales documents. References are curated. Demos use ideal inputs. None of these reliably predict whether an engagement will go well.
The six dimensions below give you a more diagnostic framework — one that surfaces how a vendor actually operates, not just how they present themselves.
Dimension 1: Eval maturity
What it is: The degree to which a vendor measures whether their AI systems are working correctly and can demonstrate improvement over time.
This is the most important differentiator between vendors who have shipped production AI and vendors who have built demos. Evals — structured test suites that measure model or system performance — are the foundation of any AI engineering practice. Without them, optimization is guesswork and regressions are invisible.
What to ask:
- "Do you know what Ragas is?" (A widely-used RAG evaluation framework. A "yes" doesn't prove expertise, but a blank stare suggests limited production RAG experience.)
- "How do you measure whether an agent is performing correctly?"
- "Can you show me an eval report from a previous project?"
- "What metric were you optimizing for on your last AI project, and what was the final number?"
What good looks like: The vendor can describe a specific eval approach — an offline test set with N examples, a metric like retrieval recall or task completion rate, a target threshold, and a process for re-running evals when the model or system changes. They have eval reports from previous projects they can share (with client permission or anonymized).
What a red flag looks like: "We test it thoroughly before delivery." "We do extensive QA." No named metrics, no eval tooling, no reports. This is not an engineering practice — it's intuition dressed up as process.
Dimension 2: Code ownership
What it is: Whether you receive the code, can run it without the vendor, and are free to take it elsewhere from day one.
This is non-negotiable for any engagement where the output is running software. Code ownership problems surface in several ways:
- The code lives in the vendor's GitHub organization, and you need their cooperation to access it
- The system depends on a proprietary framework the vendor built that you're licensing, not owning
- The vendor hasn't documented dependencies, so handing it off is practically impossible even if you legally own it
- "Final handoff" is a milestone in the project, meaning you don't have code until then
What to ask:
- "From day one, does the code live in our version control?"
- "Does the system depend on any proprietary tooling or frameworks you've built?"
- "If we end this engagement today, can we hand this code to a different developer without your help?"
- "What does the handoff documentation look like?"
What good looks like: Code lives in your repository from the first commit. No proprietary dependencies that create lock-in. The vendor writes documentation as they go, not in a last-week scramble. A knowledgeable developer who wasn't involved in the project could read the codebase and understand it.
What a red flag looks like: "We'll deliver the final codebase at the end of the project." "Our framework handles X — you'll need a license for that." "Documentation is included in the final deliverables." Any answer that treats code access as a milestone rather than a default.
Dimension 3: Pricing model
What it is: How the vendor structures their fees and what triggers cost increases.
No pricing model is inherently better — fixed scope, retainer, and time-and-materials all work in the right context. What matters is whether the pricing model aligns incentives correctly for your situation.
Fixed scope: You agree on a spec, a price, and a timeline. Vendor bears the risk of underestimating scope. Good when you have a well-defined output and don't expect significant changes during the build. Risk: vendor cuts corners when they've underestimated.
Retainer (monthly): Fixed monthly fee for a defined set of deliverables or capacity. Good for ongoing work where scope evolves. Risk: deliverables become vague over time, value drift.
Time and materials (T&M): You pay for hours worked. Good when scope is genuinely unknown. Risk: hours expand without clear output accountability.
What to ask:
- "What happens if we need to change the scope after the project starts?"
- "How are change orders handled and priced?"
- "What's included in your monthly retainer and what triggers out-of-scope billing?"
- "Can you give me a ballpark range before we do a discovery process?"
What good looks like: Clear, written scope with documented change-order triggers. Transparent out-of-scope pricing. A vendor willing to give you an order-of-magnitude estimate before a formal scoping process.
What a red flag looks like: Vague scope with lots of "as needed" language. Hourly rates with no ceiling or milestone gates. Change orders that are not priced in the contract — they just "appear on the invoice." Refusal to name any number until after a paid discovery engagement.
Dimension 4: Security posture
What it is: Whether the vendor handles your data responsibly and can articulate their security practices clearly.
This matters more for AI projects than for typical software development because AI pipelines often process sensitive data — customer records, proprietary content, financial information, health data. A vendor who can't answer basic security questions is a vendor whose systems you can't trust with sensitive inputs.
What to ask:
- "Can you sign an NDA before we share technical context?"
- "Where is the code hosted during development? Who has access?"
- "How do you handle customer data that appears in prompts or fine-tuning datasets?"
- "What happens to data after the engagement ends — is it deleted?"
- "Do you have a SOC 2 report or equivalent?"
What good looks like: The vendor signs an NDA without pushback. They can explain their data handling practices for development and production. They don't send customer data through personal accounts or unsecured tools. They have a clear answer for what happens to your data at engagement end.
What a red flag looks like: Hesitation to sign an NDA. Vague answers about data handling. No clear data lifecycle policy. "We use the best tools for the job" when asked about security, without specifics.
For regulated industries: ask explicitly whether they've worked with HIPAA, PCI-DSS, or similar frameworks. Inexperience with regulated data is not disqualifying, but lying about it is.
Dimension 5: Communication style
What it is: How the vendor communicates during the engagement — not how they communicate during sales.
The sales process is always better than the project process. But there are signals you can extract before signing.
Async-first vs. meetings-heavy: Async-first vendors communicate in writing, have clear written handoffs, and make decisions without requiring a synchronous meeting for every choice. This is much more efficient for technical work. Meetings-heavy vendors generate communication volume but not necessarily information value.
What to ask:
- "What does a typical week of communication look like during an active build?"
- "How do you handle blockers or questions that come up between scheduled check-ins?"
- "What's your target response time for messages on Slack/email?"
- "What do you provide in weekly status updates?"
What good looks like: A defined async update cadence (daily standup in a shared channel, weekly written status report with what was done, what's next, what's blocked). A clear escalation path for urgent issues. Written decisions that don't require a meeting to reference.
What a red flag looks like: "We have great communication, we stay very connected." "We'll set up regular syncs." Promising communication quality without describing the mechanism. Vendors who answer communication questions with personality descriptions rather than process descriptions.
One reliable test: how quickly and clearly do they respond to your pre-engagement emails? Slow, vague, or poorly organized replies during sales predict worse communication under project pressure.
Dimension 6: Post-launch plan
What it is: Whether the vendor has a clear answer for what happens after the system ships.
AI systems require ongoing maintenance in ways that traditional software does not. Models change. Providers deprecate versions. Prompt behavior drifts as the model's RLHF tuning changes over time. Retrieval quality degrades when new documents don't match the original embedding distribution.
A vendor who treats "launch" as the finish line has not shipped many production AI systems.
What to ask:
- "What's your standard post-launch support arrangement?"
- "Who owns prompt maintenance when the LLM provider updates the model?"
- "What's the SLA for issues in production after handoff?"
- "Do you maintain systems after launch, or do you hand off fully?"
- "What does a knowledge transfer look like — can your internal team maintain this without you?"
What good looks like: A defined post-launch support window (at minimum 30 days of bug-fix support after launch). A clear handoff process that includes runbooks for common issues, documentation for how to update prompts, and a list of monitoring alerts to watch. A realistic answer about what ongoing maintenance costs.
What a red flag looks like: "We'll be available if anything comes up." No defined SLA. Handoff documentation that's an afterthought. The implicit assumption that the system will just work without intervention after launch.
Putting the dimensions together
These six dimensions form a quick scoring matrix. Run through them in your first one or two conversations with a prospective vendor:
| Dimension | Question to ask | Red flag | |---|---|---| | Eval maturity | Can you show me an eval report? | No named metrics, no tooling | | Code ownership | Is the code in our repo from day one? | Final handoff, proprietary dependencies | | Pricing model | How are change orders handled? | Vague scope, no ceiling on T&M | | Security posture | Can you sign an NDA today? | NDA hesitation, vague data handling | | Communication | What does a typical week look like? | Process-free answers about "staying connected" | | Post-launch | Who owns prompt maintenance after launch? | "We'll be available if something comes up" |
A vendor who clears all six is unusual but they exist. A vendor who fails three or more is telling you something important about how the engagement will go.
The bottom line
Vendor evaluation before an AI engagement is not about finding the best demo or the most impressive proposal. It's about finding a team whose operational practices are sound enough to ship something that keeps working after you've signed off.
The six dimensions — eval maturity, code ownership, pricing model, security posture, communication style, and post-launch plan — map directly to the six most common failure points in AI development engagements. A vendor who can answer all six clearly has almost certainly shipped real AI systems. A vendor who struggles with two or three has probably shown more demos than they've shipped systems.
If you're evaluating partners for an AI project and want to understand how we approach each of these dimensions, see our AI agent development services page for specifics on how we structure our engagements.