AI agent evaluation in Seattle.

The technical reality of enterprise agent evaluation in Seattle.

Questions from Seattle buyers.

What does a serious prompt-injection test suite contain, beyond the public jailbreak lists?

Payloads built against your specific attack surface, because generic jailbreaks test the model while your risk lives in the integration. The suite structure: public corpora as the baseline (they catch regressions and known classes), then targeted payloads crafted from your architecture: injections embedded in the documents your RAG retrieves (the attacker writes the wiki page your agent reads), instructions hidden in data fields your tools return (a CRM note that says 'ignore previous instructions'), cross-tenant probes shaped like your actual tenancy model, and exfiltration attempts aimed at your real secrets surface (system prompts, connection strings, other users' context). Each payload class gets pass-rate tracking over time, and the suite runs on every prompt, model, retrieval, or tool change, because injection resistance is a property of the whole pipeline, not the model. The deliverable doubles as the artifact enterprise security reviews increasingly request by name: documented injection testing with results, methodology, and re-test cadence.

Our agent calls internal tools and APIs. How is tool-call correctness actually tested?

In a sandboxed replica where wrong calls are free, against suites that score the call, not just the conversation. The harness: your tools mocked with realistic schemas and failure modes (timeouts, permission errors, empty results, malformed responses), scenario suites replaying recorded production traces plus synthetic edge cases, and scoring at the call level: right tool selected, arguments correct and well-formed (the difference between get_invoice(id) and get_invoices(customer) is the difference between an answer and a data leak), destructive actions gated behind confirmation patterns, retries bounded, and failure recovery sensible (a tool error should produce a graceful path, not a hallucinated success). Trajectory metrics sit on top: steps versus a reference path, permission boundaries respected throughout, clean termination. For Seattle platforms where agents touch order systems, infrastructure APIs, or customer data, these gates are what separate a demo from something your SRE organization will allow within reach of production credentials.

What evaluation evidence do enterprise customers actually require before deploying our AI feature?

Increasingly, a specific artifact set, and arriving with it pre-assembled shortens procurement by weeks. The recurring asks from enterprise AI reviews: documented intended use and limitations (what the feature does, what it refuses), quality metrics on representative data with methodology (not marketing numbers: eval-set construction, judge calibration, thresholds), safety and security testing results (injection suites, data-boundary tests, refusal behavior), monitoring and incident posture (how regressions get caught in production, what triggers rollback, who gets paged), and change management (what re-tests when models or prompts change, how customers learn of material changes). Some buyers map these to frameworks (NIST AI RMF vocabulary appears more each quarter); most just want competent answers in writing. We assemble the package from the evaluation work itself, so it is evidence rather than prose, and version it so the answers stay current as the system evolves, which is what the second procurement cycle checks.

How does eval telemetry integrate with our existing observability and SIEM stack?

As structured events on the pipelines you already operate, so AI quality becomes another monitored property of the platform. The integration: every evaluated interaction emits a structured event (scores by dimension, judge version, model version, feature tag, trace ID correlating to your distributed tracing), production sampling writes to the same stream, and the events land where your platform team already lives: OpenTelemetry-compatible export, CloudWatch or your metrics store for dashboards and alarms, and the SIEM for the security-relevant subset (injection-suite failures, data-boundary violations, anomalous refusal-rate shifts read as possible probing). Alert routing follows your existing ownership model: quality regressions page the feature team, security-class events page security. The design principle is that AI evaluation should not stand up a parallel observability empire; it should make model behavior visible in the panes of glass your organization already trusts, with the same severity discipline and the same on-call expectations.