AI agent evaluation in Miami.

The technical reality of multilingual evaluation in Miami.

Questions from Miami buyers.

How do we build evaluation sets for Spanish and Portuguese that are not just translated English tests?

From native traffic, labeled by native speakers, with failure modes scouted per language. Translated test sets inherit English's structure and miss exactly what matters: the idioms customers actually type, regional register differences (Caribbean Spanish support messages read differently than Rioplatense ones; Brazilian Portuguese is not European Portuguese with a tan), and the formality calls (tú/usted, você/o senhor) where a wrong choice reads as rude or absurd. The construction we run: mine each language's production logs separately, cluster by intent within the language, sample to that language's own distribution (intent mixes differ by market, not just words), and label with native-speaker graders using rubrics localized in substance, not just translation: the Spanish rubric carries its own register guidance. Where one language's volume is thin, we say so and supplement with native-authored synthetic cases reviewed by your bilingual staff, flagged as synthetic in the set so the provenance stays honest.

How do native-speaker grading operations actually work without becoming a staffing project?

Like clinician grading in healthcare evals: concentrated, calibrated, and then leveraged. The workflow: your bilingual staff (support leads, market managers) grade an initial 150 to 250 items per language in structured sessions (60-to-90-minute blocks, an engineer present, rubric disputes resolved live and written down), which establishes the human standard at a cost of one to two workdays per language. LLM judges, prompted per language with the localized rubric and graded examples, then carry the volume, with agreement against the human standard measured and reported before their scores count. Monthly maintenance is a sampled re-grade (20 to 30 items per language) that keeps the judges anchored and catches rubric drift. Where your team lacks a language entirely, we arrange vetted professional graders for the standard-setting phase, but the durable design keeps your own market staff as the quality anchor, because they hear the customers the eval is meant to protect.

Our customers code-switch mid-message: Spanish and English in one sentence. Does evaluation cover that?

It must, because code-switched traffic is where multilingual systems fail quietest. Miami's reality (Spanglish support messages, Portuguese emails with English product terms, English chats that pivot to Spanish at the emotional moment) sits outside most teams' eval coverage because their sets are monolingual by construction. The coverage we build: a code-switching slice mined from real logs (these messages are identifiable by language-detection entropy), labeled for the behaviors that matter: did the system understand the mixed input, did it respond in the right language (the customer's dominant language, or the language of the question's operative clause; your CX team sets the policy and the rubric encodes it), and did terminology survive the switch (the English product name inside a Spanish sentence should not get translated into nonsense). Failure rates on code-switched slices routinely run two to four times the monolingual rate, which is precisely why the slice gets its own dashboard line rather than vanishing into a language average.

Policies differ by market: what is compliant in the US differs from Brazil or Mexico. How does evaluation handle that?

With per-market policy suites that test the system against each jurisdiction's rules, not a lowest-common-denominator blend. The structure: your compliance and market teams supply the per-market rule set (disclosure language for financial products, marketing claim restrictions, data-handling disclosures under LGPD versus US state laws, refund and cancellation rights that differ by country), and each rule becomes test cases run against the market-appropriate configuration: the Brazil-facing bot gets tested against Brazilian consumer-rights scenarios in Portuguese, the Mexico flow against Profeco-shaped expectations in Spanish. Cross-market leakage gets tested explicitly (the US policy answer surfacing for a Brazilian customer is its own failure class, and a common one when systems share prompts across markets). Results report per market and per language, because 'compliant on average' is not a thing any regulator recognizes. The suite re-runs when either the system or any market's rules change, with rule-library versioning so your compliance team owns the source of truth.