AI agent evaluation in Boston.

The technical reality of clinical AI evaluation in Boston.

Questions from Boston buyers.

How do we build a clinical eval set without consuming a clinician's month?

By spending clinician hours only where clinical judgment is irreplaceable, and engineering everything around them. The workflow: we draft the candidate set from real de-identified cases (sampled across encounter types, specialties, and the messy edges like polypharmacy and ambiguous abbreviations), pre-screen with non-clinical labelers for the mechanical properties (format, completeness, source linkage), and bring clinicians a focused grading task with a rubric co-written in their language: not 'rate 1 to 5' but 'would this summary mislead a covering physician.' Grading sessions run in 60-to-90-minute blocks with an engineer present so rubric ambiguities resolve immediately rather than by email. A typical clinical eval set (200 to 300 graded items) costs a clinician eight to twelve hours total, after which calibrated LLM judges carry the volume with monthly sampled re-verification against the clinician standard. The set then appreciates: every production correction a clinician makes gets sampled back in, so the ground truth compounds instead of staling.

What does refusal and escalation testing look like for a patient-facing or clinician-facing feature?

A test battery for the moments that matter most and occur least. We build suites for three behaviors: hard refusals (the system must not answer: diagnosis requests in an administrative tool, medication-change advice in a scheduling bot), escalations (the system must hand off: red-flag symptom language in a portal message, suicidality signals, anything where delay has clinical cost), and graceful boundaries (the system answers the safe part and routes the rest). The suites combine curated cases (clinician-identified red flags), adversarial variants (the same red flag phrased casually, misspelled, buried mid-paragraph, or expressed in the second person about a child), and multi-turn erosion (users who push back). Pass criteria are strict: escalation suites typically gate at or near 100 percent, because a single missed red flag is the failure mode that ends programs. These suites run on every prompt and model change, since refusal behavior is precisely what shifts silently when anything upstream moves.

How is hallucination actually measured in clinical summarization, beyond anecdotes?

Claim-level verification against the source record, with error taxonomy that distinguishes the dangerous from the cosmetic. The harness decomposes each generated summary into atomic clinical claims (medication X at dose Y, finding Z present, plan A agreed) and verifies each against the source encounter: supported, unsupported (present in the summary, absent in the record: the classic fabrication), contradicted (summary says discontinued, record says continued: the dangerous class), or omitted (record-critical fact missing from summary, which clinical reviewers often rate as the worst failure). LLM judges do the volume after calibration against clinician-graded samples, and results report rates per error class per note type, because a 2 percent fabrication rate concentrated in medication lists is a different emergency than the same rate in social history. Thresholds then gate by stakes: discharge-adjacent summaries hold the strictest bars. This produces the number your clinical governance committee actually needs: not 'does it hallucinate' but 'at what rate, in which class, on which content.'

When does evaluation evidence start mattering for the FDA SaMD question?

Earlier than most teams expect, and in a useful way regardless of where you land. If your feature stays administrative (scheduling, documentation support, billing workflows), rigorous evaluation evidence documents exactly why: intended use, the behaviors tested, the refusal suites proving the system declines clinical-decision territory. That artifact is what counsel uses to assess the device boundary with confidence rather than instinct, and it is what an institutional review committee reads before a pilot. If your roadmap crosses toward clinical decision support, the evaluation infrastructure you built becomes the foundation of a regulatory strategy: predefined performance metrics, locked test sets, change-control on models and prompts, all things FDA-facing development requires anyway. We are engineers, not regulatory counsel, and we say so; what we build is the evidence layer that makes counsel's job, and any future submission, tractable. Teams that defer evaluation until the regulatory question forces it pay twice: once for the rush, once for the rebuild.