AI agent evaluation in Phoenix.

The technical reality of operations-AI evaluation in Phoenix.

Questions from Phoenix buyers.

Our drafting pipeline produces forty letter classes. How granular should accuracy measurement be?

Per letter class, because the classes fail differently and carry different stakes. A blended 'drafting accuracy' number averages a payoff-quote letter (where a wrong figure is a legal event) with a payment-received acknowledgment (where the worst error is awkward phrasing), and that average guides nothing. The harness we build: each letter class gets its own labeled sample (drawn from real account records including the edge cases: bankruptcies, deceased borrowers, partial payments, attorney-represented accounts), its own correctness criteria (figures exact, required language present, account facts faithful), and its own threshold tied to its consequence tier. Reporting then shows the distribution: which classes are auto-send ready, which need review, which the model should not draft yet. The practical discovery in most Phoenix audits: a handful of classes drive most of the risk while the long tail is benign, which means review capacity concentrates where it matters and the benign tail earns automation months sooner than a blended number would have allowed.

What does examiner-grade sampling mean for our QA, practically?

Sampling designed so its conclusions survive someone whose job is to doubt them. The properties: documented methodology (stratification by letter class, state, and account type, with the rationale written down), statistically defensible sizes (calculated from the error-rate confidence you need per stratum, not from what the QA team had time for), blind protocols where grading bias matters (reviewers score drafts without seeing the model's confidence or which system produced them), and reproducibility (the sample frame, selection seed, and grading rubric versioned, so the same audit can be re-run and lands in the same place). The output is an error-rate statement with confidence bounds per stratum, the kind of sentence that holds up in a regulator's interview or a vendor-management review: 'monthly stratified sample, n per class, error rate X percent with a Y-percent upper bound at 95 percent confidence.' Most internal QA programs have the effort but not the design; the redesign usually costs nothing extra in reviewer hours and converts the same labor into defensible numbers.

How should auto-send thresholds be governed once they are set?

Like credit policy: owned, versioned, monitored, and revocable by rule rather than by meeting. The governance frame we install: each letter class's auto-send threshold is a written policy entry (the threshold, the evidence basis, the owner, the review date), production performance is monitored per class against control bands (error rates from sampling, recall rates from downstream corrections, complaint signals), and breaches trigger automatic demotion: a class whose measured error rate exits its band drops back to mandatory review without waiting for a quarterly meeting, with the alert routed to the policy owner. Re-promotion requires evidence, not elapsed time: a refreshed sample demonstrating the band is restored. Changes upstream (template revisions, model swaps, rule-library updates) automatically demote affected classes to review until their suites re-pass. This converts the scariest question in drafting automation ('what if it degrades and nobody notices') into a tested property: degradation gets noticed by the system that demotes first and pages second.

What evaluation records should we be keeping for audits, and for how long?

Enough to reconstruct any output's quality context, retained on the schedule your correspondence already follows. The record set we build: per-run harness results (eval set version, model and template versions, scores, pass/fail against thresholds) for every change-triggered and scheduled run; per-letter lineage linking each sent letter to the policy version, rule checks passed, model, and auto-send-versus-reviewed path it took; sampling records (methodology, samples, grades, error-rate statements) as the ongoing quality evidence; and threshold-policy history (who set what, on what evidence, when it changed). Retention follows your correspondence retention schedule, since the evaluation context of a letter is only useful as long as the letter itself can be questioned. Storage is rounding error (structured JSON in object storage), and the retrieval design matters more than the volume: an auditor asking about a specific letter from last March should be answerable in minutes. We build the records as a by-product of the harness, because evidence that requires separate effort stops being kept the first busy quarter.