AI agent evaluation in Denver.

The technical reality of lean-team evaluation in Denver.

Questions from Denver buyers.

What is the minimum viable eval for a two-engineer team, honestly?

Three components, maintainable in an afternoon a month. One: a labeled set of 100 to 200 real cases (mined from production logs, labeled by whoever owns quality, refreshed monthly with new failures), small enough to stay current and large enough to catch real regressions. Two: a smoke suite in CI (the 30 highest-signal cases, run on every prompt or model change, failing loudly on format breaks, refusal failures, or score drops beyond noise). Three: a weekly production sample (twenty random outputs eyeballed by a human against a one-page rubric, which catches the drift no automated judge notices because the rubric itself needs updating). That is the floor, and it is genuinely enough for most Denver-stage products: it converts 'the AI seems fine' into a trend line, and it costs hours, not headcount. Everything beyond it (calibrated judges, trajectory testing, per-slice dashboards) gets added when a specific stake justifies it, not because a blog post listed it.

Our content has hard compliance rules per state. Can those be tests instead of reviewer memory?

Yes, and rule-shaped requirements are the easiest evaluation wins available. Compliance rules (cannabis marketing restrictions by state, financial-product disclosure requirements, energy-lease notification language) decompose into testable assertions: prohibited terms and claims per jurisdiction (string and pattern checks, no model needed), required disclosures present and correctly worded (deterministic checks against your rule library), and contextual rules (claims that need qualifiers, audience restrictions) handled by a judge prompted with the specific rule text and calibrated against your compliance reviewer's grades. The suite runs on every generated piece before it ships, per jurisdiction, with failures routed to review pre-publication rather than discovered post-complaint. The honest division: the rule layer catches violations of written rules at near-perfect rates because most are mechanical; your compliance owner keeps judgment calls and rule-library updates. Teams typically convert their reviewer from reading everything to reading flagged items plus a sample, which is the same trust shift that made code review survivable.

How much QA sampling does our document-extraction pipeline actually need?

Less than blanket review and more than spot checks, with the rate set by measured risk rather than anxiety. The statistics are friendly here: to detect whether field-level error rates have drifted beyond an acceptable threshold, you need a fixed sample per period that depends on the threshold and confidence you want, not on volume; a few hundred sampled documents a month gives tight confidence on error rates even at large volumes. The design we install: stratified sampling (more weight on document types and sources with historical errors, new vendors, and recent pipeline changes), exception-queue exclusion (those already get human eyes; sampling covers the auto-posted majority where silent errors hide), and corrections fed back into the labeled set so the eval ages with the document mix. The output is a monthly error-rate dashboard with control bands: in-band means the sampling continues, out-of-band triggers investigation. The cost is a few reviewer-hours monthly, which is what staffing a real number costs.

When is more evaluation investment actually worth it for a team our size?

When a specific event raises the stakes, and the triggers are recognizable in advance. Worth the upgrade: an enterprise customer's security or quality review (their questionnaire will ask for methodology; a calibrated harness answers it), an agent gaining write access to real systems (trajectory and tool-call testing stop being optional when mistakes touch production data), a regulated surface (anything generating compliance-relevant content per jurisdiction deserves the rule suite plus audit-ready records), a containment or accuracy number being quoted to investors or boards (numbers that get quoted get audited eventually), and post-incident (the cheapest time to build the harness that would have caught it). Not yet worth it: pre-launch products with no real traffic (build the starter set, defer the machinery), and features where a human reviews every output anyway (the review is the eval; instrument it instead). We tell Denver teams which side of the line they are on in the scoping call, and the answer is sometimes the smaller engagement.