AI Agent Evaluation · San Francisco, CA

AI agent evaluation in San Francisco.

San Francisco ships AI products faster than anyone, and the ones that keep winning share a discipline: they measure. Eval sets built from real traffic, judges calibrated against human standards, regression gates that catch the Friday-night prompt tweak, and model swaps treated as routine experiments rather than acts of faith.

We build that machinery: first eval sets mined from your logs, LLM-as-judge calibration with provenance, tiered CI gates, trajectory testing for agents, and the eval-driven habits that make iteration safe at SF speed.

How we work

Tell us the product and what quality means for it.

The technical reality of evals in SF product teams.

First eval sets come from logs, not imagination: clustered production traffic sampled to mirror reality, failure cases from tickets, and 150 to 300 items labeled by the person whose taste is the product. Small, fresh, and maintained beats large, imagined, and stale every quarter of the year.

LLM judges scale measurement only after calibration: human grades first, judge tuned to demonstrated agreement, scores shipped with provenance and re-verified monthly. A judge score without calibration is a vibe wearing a decimal point, and investors are learning to ask.

CI gates earn their keep by being tiered and humane: smoke suites in minutes on every PR, full runs nightly with per-slice deltas, statistical thresholds that fail on signal rather than sampling noise, and an override path so the gate survives its first urgent Friday.

Model swaps become quarterly opportunities once the harness exists: per-slice comparison, cost and latency beside quality, canaried rollout, boring rollback. The eval set compounds into the one moat an AI product genuinely owns.

What an evaluation engagement includes.

The standard build, tuned for fast-moving product teams.

Log-mined eval sets

Clustered, proportionally sampled, failure-enriched suites labeled with your quality owner, sized to maintain rather than admire.

Judge calibration with provenance

LLM judges tuned to demonstrated human agreement, scores carrying their calibration numbers, monthly sampled re-verification.

Tiered CI gates

Smoke on every PR, full nightly, adversarial weekly, with noise-aware thresholds, per-slice diffs, and a documented override.

Agent trajectory testing

Sandboxed replays scoring tool calls, recovery, efficiency, and stopping discipline for anything that acts rather than answers.

Model-swap runbook

Per-slice candidate comparison, three-dimensional decision dashboards, canary plans, and rollback that stays boring.

Production drift monitors

Sampled scoring of live traffic, slice-level alerts, and the feedback loop that keeps the eval set growing with reality.

Built for San Francisco's iteration speed.

SF product teams change prompts daily, swap models quarterly, and compete against last month's batch; the only sustainable way to move that fast is measurement that moves faster. Evaluation here is not a compliance artifact, it is the product discipline that separates compounding teams from thrashing ones, and increasingly the diligence question that separates funded ones from passed-on ones.

We build the machinery and the habit: harnesses your team owns, runbooks that make swaps routine, and an eval culture that survives our departure, because rented discipline is not discipline.

We work with SF teams remotely, in Pacific hours, with first harnesses typically standing in two to three weeks.

Questions from San Francisco buyers.

We have production logs but no eval set. How do we build the first one without boiling the ocean?: Mine the logs for the distribution, then label the slice that matters. The method we run in week one: cluster a few thousand recent production inputs by intent and difficulty (cheap embedding clustering does this fine), sample proportionally so the set mirrors reality rather than the team's favorite demos, add the failure cases support tickets and thumbs-down events already identified, and label 150 to 300 items with the person whose quality judgment actually counts (founder, lead, domain expert), using a rubric written during labeling rather than before it. That set is small enough to maintain and large enough to detect meaningful regressions. The classic mistakes we design around: sets built from imagined inputs (production traffic is always weirder), rubrics defined in abstract committee (they collapse on contact with real outputs), and 2,000-item sets nobody ever re-labels (a stale large set loses to a fresh small one every time).
How reliable is LLM-as-judge, really, and when can we trust it over human review?: Reliable enough to run your volume after calibration, never instead of it. The honest workflow: humans grade a few hundred outputs first, establishing the ground truth and surfacing rubric ambiguities; the judge prompt is then tuned until judge-human agreement on a held-out slice is strong (we target agreement comparable to human-human consistency on the same task, which for subjective qualities is lower than teams expect); and judge scores ship with that calibration number attached, re-verified on a sampled basis monthly. Where judges work well: faithfulness checks, rubric compliance, format and policy adherence, pairwise comparisons. Where they stay weak: subtle domain correctness (a plausible wrong answer fools them the way it fools laypeople) and anything where your taste is the product. The operating rule: judges scale the measurement, humans own the standard, and any judge score without a calibration provenance is a vibe wearing a decimal point.
What does a good regression gate in CI look like for an LLM product?: Tiered, fast where it must be, and honest about flakiness. The structure we install: a smoke tier (30 to 50 high-signal cases, run on every PR, minutes not hours, hard-failing on genuine regressions like format breaks, refusal failures, or faithfulness drops beyond noise), a full tier (the complete eval set, run nightly and pre-release, with score deltas reported by slice rather than one blended number), and an exploration tier (adversarial and long-tail suites, run weekly or on demand). The engineering details that make it humane: deterministic settings where possible, statistical thresholds tuned so the gate fails on signal rather than sampling noise (LLM outputs vary; the gate must know that), results posted on the PR with per-slice diffs so the offending change is findable, and a documented override path, because a gate without an escape hatch gets disabled the first urgent Friday.
Models change under us constantly. How does eval-driven development keep model swaps from being scary?: By making the swap a measured event instead of a leap. With a real harness in place, a model swap is a routine: run the full eval on the candidate, compare per-slice (not just the average; new models routinely improve summarization while regressing on your specific extraction format), run cost and latency beside quality so the decision is three-dimensional, canary the winner on a traffic slice with production monitors watching, and keep the rollback boring. Teams with this machinery treat model releases as quarterly opportunities; teams without it treat them as threats and calcify on aging models at premium prices. The same harness powers the rest of eval-driven development: prompt changes ship with before-and-after scores, feature ideas get judged by eval impact rather than demo applause, and the eval set itself becomes a compounding asset, the closest thing an AI product has to a moat it actually owns.

Ready to measure what you ship?

Tell us the product, the traffic volume, and what quality means to you. We reply within one business day with a scope and a fixed price.

AI agent evaluation in San Francisco.

The technical reality of evals in SF product teams.

What an evaluation engagement includes.

The standard build, tuned for fast-moving product teams.

Log-mined eval sets

Clustered, proportionally sampled, failure-enriched suites labeled with your quality owner, sized to maintain rather than admire.

Judge calibration with provenance

LLM judges tuned to demonstrated human agreement, scores carrying their calibration numbers, monthly sampled re-verification.

Tiered CI gates

Smoke on every PR, full nightly, adversarial weekly, with noise-aware thresholds, per-slice diffs, and a documented override.

Agent trajectory testing

Sandboxed replays scoring tool calls, recovery, efficiency, and stopping discipline for anything that acts rather than answers.

Model-swap runbook

Per-slice candidate comparison, three-dimensional decision dashboards, canary plans, and rollback that stays boring.

Production drift monitors

Sampled scoring of live traffic, slice-level alerts, and the feedback loop that keeps the eval set growing with reality.

Built for San Francisco's iteration speed.

We build the machinery and the habit: harnesses your team owns, runbooks that make swaps routine, and an eval culture that survives our departure, because rented discipline is not discipline.

We work with SF teams remotely, in Pacific hours, with first harnesses typically standing in two to three weeks.

Questions from San Francisco buyers.

We have production logs but no eval set. How do we build the first one without boiling the ocean?

Mine the logs for the distribution, then label the slice that matters. The method we run in week one: cluster a few thousand recent production inputs by intent and difficulty (cheap embedding clustering does this fine), sample proportionally so the set mirrors reality rather than the team's favorite demos, add the failure cases support tickets and thumbs-down events already identified, and label 150 to 300 items with the person whose quality judgment actually counts (founder, lead, domain expert), using a rubric written during labeling rather than before it. That set is small enough to maintain and large enough to detect meaningful regressions. The classic mistakes we design around: sets built from imagined inputs (production traffic is always weirder), rubrics defined in abstract committee (they collapse on contact with real outputs), and 2,000-item sets nobody ever re-labels (a stale large set loses to a fresh small one every time).

How reliable is LLM-as-judge, really, and when can we trust it over human review?

Reliable enough to run your volume after calibration, never instead of it. The honest workflow: humans grade a few hundred outputs first, establishing the ground truth and surfacing rubric ambiguities; the judge prompt is then tuned until judge-human agreement on a held-out slice is strong (we target agreement comparable to human-human consistency on the same task, which for subjective qualities is lower than teams expect); and judge scores ship with that calibration number attached, re-verified on a sampled basis monthly. Where judges work well: faithfulness checks, rubric compliance, format and policy adherence, pairwise comparisons. Where they stay weak: subtle domain correctness (a plausible wrong answer fools them the way it fools laypeople) and anything where your taste is the product. The operating rule: judges scale the measurement, humans own the standard, and any judge score without a calibration provenance is a vibe wearing a decimal point.

What does a good regression gate in CI look like for an LLM product?

Tiered, fast where it must be, and honest about flakiness. The structure we install: a smoke tier (30 to 50 high-signal cases, run on every PR, minutes not hours, hard-failing on genuine regressions like format breaks, refusal failures, or faithfulness drops beyond noise), a full tier (the complete eval set, run nightly and pre-release, with score deltas reported by slice rather than one blended number), and an exploration tier (adversarial and long-tail suites, run weekly or on demand). The engineering details that make it humane: deterministic settings where possible, statistical thresholds tuned so the gate fails on signal rather than sampling noise (LLM outputs vary; the gate must know that), results posted on the PR with per-slice diffs so the offending change is findable, and a documented override path, because a gate without an escape hatch gets disabled the first urgent Friday.

Models change under us constantly. How does eval-driven development keep model swaps from being scary?

By making the swap a measured event instead of a leap. With a real harness in place, a model swap is a routine: run the full eval on the candidate, compare per-slice (not just the average; new models routinely improve summarization while regressing on your specific extraction format), run cost and latency beside quality so the decision is three-dimensional, canary the winner on a traffic slice with production monitors watching, and keep the rollback boring. Teams with this machinery treat model releases as quarterly opportunities; teams without it treat them as threats and calcify on aging models at premium prices. The same harness powers the rest of eval-driven development: prompt changes ship with before-and-after scores, feature ideas get judged by eval impact rather than demo applause, and the eval set itself becomes a compounding asset, the closest thing an AI product has to a moat it actually owns.

Ready to measure what you ship?

Tell us the product, the traffic volume, and what quality means to you. We reply within one business day with a scope and a fixed price.