AI agent evaluation in Nashville.

The technical reality of operations evaluation in Nashville.

Questions from Nashville buyers.

How is prior-auth packet completeness actually scored, beyond 'looks assembled'?

Against the payer's stated criteria, item by item, with the gaps that cause denials weighted heaviest. The harness: for each payer-service combination, the criteria checklist (medical-necessity documentation, relevant history, prior treatment evidence, correct codes) becomes the scoring rubric; assembled packets get verified item-by-item: criterion addressed, supporting document present and correct (the right date range of notes, the right report, not just a document of the right type), and evidence linked to its source in the record. The metrics that matter operationally: completeness rate per payer and service line, the specific criteria most often missed (which drives upstream fixes in documentation workflows), and the correlation against outcomes once decisions return: packets the scorer flagged incomplete should deny at higher rates, and that correlation is the harness validating itself against ground truth. Revenue-cycle leaders use the per-payer breakdown directly: a payer whose criteria interpretation drifts becomes visible in the completeness-versus-denial gap, which is negotiation evidence no anecdote provides.

Our red-flag classifier routes risky patient messages to humans with no draft. How is that tested hard enough to trust?

Like a safety system: adversarially, asymmetrically, and on every change. The test battery: curated red-flag cases from clinical leadership (symptom language requiring escalation, medication-safety signals, self-harm indicators, anything where delay carries clinical cost), expanded systematically with the variants real patients write: misspelled, casual, buried mid-paragraph in a billing question, phrased about a family member, expressed through emoji or abbreviation, and split across a multi-message thread. The gate is asymmetric by design: misses (a red flag routed to auto-draft) gate at or near zero because that is the failure mode that ends programs and harms patients, while over-escalation (routine messages sent to humans) is tracked as an ergonomics cost, tuned only after the miss rate holds. The suite re-runs on every prompt, model, or template change, since classifier behavior shifts silently with upstream edits, and production sampling audits the auto-drafted stream continuously for red flags that slipped through, feeding any catch back into the battery permanently.

What validation should auto-send thresholds pass before routine guest or patient messages go out unreviewed?

Class-by-class evidence, staged exposure, and an automatic demotion rule, in that order. The validation sequence we run: per message class (booking confirmations, standard pre-visit instructions, refill-status notes), a labeled sample establishes accuracy against the class's correctness criteria (facts right against the record, required elements present, tone within policy); classes clearing their threshold enter a shadow phase (drafts generated and auto-send-scored but still human-reviewed, with disagreement between the score and the reviewer measured); shadow agreement holding for a defined volume earns staged auto-send (a traffic percentage, then full), with production sampling continuing permanently. The demotion rule is the trust mechanism: any class whose sampled error rate exits its control band drops back to mandatory review automatically, alert first, meeting later. Hospitality and healthcare differ only in stakes calibration: a wrong parking instruction embarrasses, a wrong medication instruction harms, so the thresholds and sampling rates tier accordingly, and the clinical classes carry the red-flag battery upstream of everything.

Leadership asks what evaluation returns in staff-hours. What is the honest accounting?

Evaluation returns hours indirectly but measurably, through three mechanisms an operations P&L recognizes. Review-economics transformation: verified drafts with flagged uncertainties convert specialist review from defensive reading (the half-hour that eats the automation's savings) into targeted checking (minutes), and edit-distance plus review-time logs prove which mode your program runs in. Automation acceleration: per-class evidence lets benign message and letter classes earn auto-send months earlier than blanket caution would allow, and the staged-validation record is what lets compliance sign off; the hours come from queues that stop requiring eyes. Incident prevention: a regression caught in CI (a template change that broke required language, a model swap that degraded packet completeness) costs minutes; the same regression caught by a payer denial wave or a patient complaint costs weeks, and the smoke-suite log quantifies how often that trade fired. The honesty rules: hours count only where queue throughput data shows them, and the evaluation program's own costs (grading sessions, sampling reviewer-hours) sit in the same ledger. We deliver that ledger quarterly, in the format your operations review already reads.