AI agent evaluation in Houston.

The technical reality of industrial-AI evaluation in Houston.

Questions from Houston buyers.

Our domain experts are drilling engineers and HSE leads, not data people. How do they grade an eval set?

Through grading sessions engineered around their time, in their vocabulary, on their documents. The workflow mirrors what works with clinicians: we draft the candidate set from real de-identified reports (sampled across rigs, basins, report formats, and the messy shorthand variants), pre-screen for mechanical properties ourselves, and bring the engineers a focused task with a rubric in their language: not 'rate the summary 1 to 5' but 'would this rollup mislead the morning meeting' and 'is the NPT attribution right.' Sessions run 60 to 90 minutes with one of us present, so when two engineers disagree about whether 'POOH for BHA change' was summarized correctly, the disagreement gets resolved and written into the rubric immediately. A working eval set (200 to 300 graded items) costs your technical staff a day or two total. Calibrated judges then carry volume, with monthly sampled re-grades keeping them anchored to the people who can actually read a morning report.

We re-scored two years of HSE incidents with the model. How do we know the classifications are consistent and right?

Two separate checks, because consistency and correctness fail independently. Consistency: the same narrative, or trivially varied phrasings of it, must classify identically; we test with paraphrase suites (real incidents reworded, reordered, abbreviated) and measure classification stability, which catches the prompt sensitivity that makes model scoring quietly arbitrary. Correctness: a stratified sample of the backfill (weighted toward severity boundaries, where misclassification matters most) gets blind-graded by your HSE leads against the matrix, and the model's agreement with that standard is the accuracy number, reported per severity tier because an error at the recordable/non-recordable boundary is not the same event as one between minor categories. The boundary cases your graders disputed among themselves get flagged as genuinely ambiguous and routed to human classification permanently, which is honest: the model should not be the tiebreaker on calls the experts split on. Recordability determinations stay human regardless; the eval proves the assembly is trustworthy, not that judgment is replaceable.

Some of these outputs feed safety-relevant decisions. How are thresholds set differently for that?

Asymmetrically and explicitly, with the failure directions priced in consequence rather than dollars. For safety-relevant surfaces (HSE classification, anomaly flagging against well plans, permit-condition extraction), the evaluation regime changes in three ways. Thresholds: miss rates on the dangerous direction gate near zero (a missed high-severity signal is the failure mode that matters; a false alarm costs minutes), and the system is tuned to over-flag rather than under-flag, with the over-flag rate managed as an ergonomics problem rather than traded against misses. Coverage: the eval set over-samples the rare-but-critical cases (the near-miss narratives, the anomalies two standard deviations out) because uniform sampling underweights exactly what safety review cares about. Change control: any prompt, model, or glossary change re-runs the full safety suite before deployment, no exceptions, with results logged for your process-safety documentation. The framing we hold throughout: the model assembles and flags, your engineers decide, and the evaluation proves the assembly never quietly drops what a human needed to see.

Our data policy keeps everything in-tenancy. Does the evaluation infrastructure live there too?

Yes, all of it, because an eval harness that exfiltrates your drilling reports to grade them defeats its own purpose. The architecture: eval sets, graded standards, and run histories live in your object storage; harness execution runs on your compute (a small container workload, scheduled or CI-triggered); judge models run through the same in-tenancy inference (Bedrock or Azure OpenAI) your production traffic uses, under the same governance; and dashboards land in your existing observability stack rather than a third-party eval SaaS. Where teams use self-hosted models for the strictest data classes, the judges run there too, with the honest caveat that self-hosted judge quality gets verified against your graded standard like any other judge. The deliverable includes the infrastructure-as-code for the whole harness, so your platform team owns it outright, and the only thing that leaves your tenancy is the invoice. This question disqualifies most off-the-shelf eval platforms for Houston operators, which is usually why the conversation reaches us.