AI agent evaluation in Dallas.

The technical reality of regulated-AI evaluation in Dallas.

Questions from Dallas buyers.

How do we test that FDCPA and state-law rule checks actually catch violations before letters go out?

By attacking your own rule layer with a violation corpus, the way security teams attack their own firewalls. The harness: a test corpus of known-violating drafts (prohibited phrasing variants, missing state disclosures, contact-frequency conflicts, dispute-status violations), built from your compliance team's enforcement knowledge plus systematic mutation (the prohibited phrase rephrased, abbreviated, embedded mid-paragraph, or split across sentences), run against the rule layer with the expectation that every violation gets caught and correctly classified. The metric is recall on violations, and the bar is set near 100 percent because a leaked violation is a regulatory event, not a quality blemish. False-positive rates get measured too (over-blocking buries reviewers and breeds workarounds), but the asymmetry is explicit. The suite re-runs on every rule-library and template change, and the violation corpus grows with every real-world catch, which converts each near-miss into permanent coverage. This is the difference between 'we have rule checks' and 'our rule checks are tested,' and examiners increasingly know to ask which one you mean.

Underwriters rely on AI file summaries. How do we verify the summaries are faithful to the file?

Claim-level verification against the source documents, with the error taxonomy underwriting actually cares about. The harness decomposes each summary into atomic assertions (loss history counts, payroll figures, operation descriptions, prior carrier facts) and verifies each against the submission documents: supported, unsupported (asserted but absent: the fabrication class), contradicted (summary says three losses, loss runs show five: the dangerous class for pricing), and omitted (material facts present in the file but missing from the summary, which underwriters often rate as the worst failure because it shapes the decision silently). LLM judges carry the volume after calibration against underwriter-graded samples, and results report per error class and per document type, so 'the summarizer is good' becomes 'fabrication under 1 percent, contradiction near zero, omission concentrated in loss-run gaps over five years old.' Thresholds gate by stakes, with bind-relevant summaries holding the strictest bars, and the underwriter's verification click (source-linked claims) stays in the product because evaluation complements review rather than replacing it.

What does a state DOI examination expect to see about our AI usage, and how does evaluation prepare us?

Examiners are converging on a recognizable artifact set, shaped by the NAIC model bulletin most states have adopted in some form: an inventory of AI systems with their intended uses, documented governance (who owns each system, how changes are approved), evidence that outputs affecting consumers are tested for accuracy and compliance (this is where evaluation records become the exhibit: rule-check recall results, faithfulness rates, bias considerations where underwriting-adjacent), audit trails connecting individual outputs to the model and rule versions that produced them, and remediation records showing that caught issues led to fixes. The preparation evaluation provides: every harness run is logged with versions and results, the violation corpus and its recall history document diligence, and the per-letter audit lineage (template version, rule checks passed, model used) means any examined letter can be reconstructed. Examiners respond to reconstructability; the deliverable is built so a sampled letter from eleven months ago answers cleanly.

Every template and policy update risks breaking something downstream. How does regression testing handle template change?

By treating templates as versioned code with a test suite, because that is what they functionally are. The machinery: each template version carries its own expected-behavior suite (required disclosures present for each state variant, prohibited phrasing absent, merge fields populating correctly against edge-case records: hyphenated names, foreign addresses, zero balances, deceased-borrower flags), and a template change triggers the full suite against the new version plus a diff report showing what changed in generated output across a representative record sample. The failure modes this catches monthly in practice: a reworded paragraph that dropped a state-required sentence, a merge-field reorder that breaks on records with missing middle names, and the classic, a fix for one state's requirement that silently changed every other state's letter. Rule-library updates get the same treatment via the violation corpus. The cultural shift is the point: template editors get CI feedback like developers do, compliance sign-off reviews a diff rather than re-reading everything, and 'small wording change' stops being a category of incident.