AI agent evaluation in Toronto.

The technical reality of governed evaluation in Toronto.

Questions from Toronto buyers.

What evaluation artifacts does OSFI's E-23 guideline expect for generative AI systems?

E-23 treats models as risk-managed assets across their lifecycle, and for generative systems the evaluation artifacts that satisfy it map cleanly: a model inventory entry with documented intended use, scope, owner, and limitations; evaluation evidence proportionate to the system's risk rating (representative test sets from your real traffic, metrics tied to the harms that matter for the use case, results against approved thresholds with sign-off); ongoing monitoring with defined triggers (drift detection on inputs and output quality, with documented escalation when bands breach); and change management (what re-evaluates when models, prompts, or data change, with records). The generative-specific translation we build: behavioral test suites instead of backtests, faithfulness and refusal metrics where decisions are assisted, and trajectory checks for anything agentic. The practical advice from Toronto engagements: write the artifacts in your model-risk team's existing vocabulary and templates; validation goes months faster when the evidence looks like evidence they already know how to challenge.

How is English/French parity actually measured, beyond running the same tests twice?

With per-language test sets built from native traffic, graded by native speakers, and compared on aligned dimensions rather than raw scores. The construction: French test sets mined from real French traffic (not translated English, which inherits English structure and misses the idioms, register choices, and régime-specific vocabulary where systems actually fail), labeled by francophone graders with rubrics localized in substance (the French rubric carries its own formality guidance: vous as default in institutional contexts, terminology aligned to Canadian usage, REER not plan d'épargne-retraite individuel borrowed from France). Parity then gets reported dimension-by-dimension: faithfulness, completeness, register correctness, refusal behavior, each per language with confidence bounds, because a system can match on accuracy while failing French register, and a blended score hides exactly that. For Quebec-facing deployments, the French suite also carries the Law 25 and Charter-relevant cases. The deliverable institutional leadership wants: a parity dashboard where a French regression pages someone before a customer posts it.

Can evaluation infrastructure run entirely in Canadian regions alongside our resident workloads?

Yes, and it should, because evaluation traffic is your production data wearing a test harness. The architecture: eval sets, graded standards, and run histories in Canadian-region object storage; harness execution on in-region compute; judge models served through the same in-region inference your production uses (Bedrock Canada Central, Azure Canadian regions), under the same governance and the same data-flow documentation; and dashboards in your existing observability stack. Where the in-region judge lineup trails the newest releases, the judge gets verified against your graded human standard like any other (measured agreement is the requirement; recency is not), and the verification record answers the question before model-risk asks it. The same documentation discipline applies: the evaluation system gets its own data-path entry in the diagrams your privacy office maintains, which takes an hour and prevents the awkward discovery that the quality tooling became the unreviewed data path. Nothing about rigorous evaluation requires leaving the region; it requires designing for it, once.

Our model vendor updates models on their schedule, not ours. How does evaluation make that manageable?

By converting vendor changes from surprises into a tested intake process. The machinery: a pinned-version policy (production runs named model versions, never floating aliases, so changes happen when you decide), a candidate-intake runbook (vendor announces a new version; the full evaluation battery runs against it: quality per slice and per language, faithfulness, refusal behavior, latency and cost, with results compared against the incumbent's baseline), a decision record (adopt, defer, or adopt-with-mitigations, signed by the system owner, filed with the model inventory), and staged rollout with production monitoring for what offline suites miss. Deprecation pressure gets managed the same way: when a vendor sunsets your pinned version, the runbook compresses but the steps hold, and the evaluation history means you know your sensitivity to change classes in advance. For OSFI-supervised institutions this is change management E-23 already expects; for everyone else it is the difference between reading vendor release notes with interest versus dread. The runbook plus your harness makes quarterly model churn a routine afternoon, which in 2026 is its own competitive posture.