AI agent evaluation in Portland.

The technical reality of brand-AI evaluation in Portland.

Questions from Portland buyers.

How does a brand voice become a rubric without flattening what makes it ours?

By extracting the rules from the examples rather than imposing rules on them. The construction process: your creative lead pulls twenty pieces they consider perfectly on-voice and ten they rejected, and we interview the differences: which words appear and which never would, how sentences breathe (length variance, rhythm, where fragments are allowed), how the brand handles claims (asserts, suggests, understates), how it addresses the reader, and what it does at emotional moments (a Portland outdoor brand and a Portland skincare brand both have warmth; they spend it differently). Those observed properties become the rubric: vocabulary lists, register parameters, claim posture, and the graded example bank carrying the cases rules cannot capture. The test of success is reversed from what teams fear: a good rubric does not make everything sound the same; it makes deviations detectable, which protects the voice when volume scales past what the creative lead can read. The lead stays the court of appeal, grading monthly samples that keep the standard anchored to taste rather than to last year's rubric.

Our products make environmental and material claims. How does evaluation verify generated copy never overstates them?

With claim-source verification as a hard pre-publication gate. The architecture: your substantiated claims live in a structured library (each claim with its evidence: certification, test result, supplier attestation, and its approved phrasings with the qualifiers that keep them accurate); generated copy gets decomposed into claims; and every detected claim must trace to a library entry, with unsupported claims and qualifier-dropping flagged before anything ships. The failure modes the gate is built for: drift from 'recycled content' to 'made from recycled materials' (a quantity claim sneaking in), 'certified' attaching to the wrong certification scope, comparative claims ('greener than') the library never substantiated, and absolute claims ('non-toxic', 'sustainable') that regulators and watchdogs read most aggressively. The suite re-runs on every prompt change and library update. For Portland brands whose credibility is the moat, this gate converts greenwashing risk from a copywriting vigilance problem into a tested property, and the library itself becomes the canonical asset your legal and sustainability teams co-own.

What does a lightweight eval stack look like, concretely, for a team without ML engineers?

Three files and a habit, honestly. The stack we install for lean Portland teams: a labeled example set (a spreadsheet-grade artifact: inputs, expected properties, grades, maintained by whoever owns quality), a runner script (a few hundred lines your engineers read in one sitting: runs the set against your prompts, calls a judge model with your rubric, outputs a scored report and per-case diffs), and a CI hook (the high-signal subset runs on prompt changes; the full set runs weekly on a schedule). No platform subscription, no dashboard service, no infrastructure beyond what already deploys your product. The habit is the real component: failures get looked at weekly, new failure types get added to the set, and the rubric gets a line edit when grading disagreements surface. Teams consistently overestimate the machinery evaluation needs and underestimate the discipline; the stack above runs on the discipline of one attentive person a few hours a month, and it is the version that survives in companies where nobody's title says evaluation.

How much human review do our generation pipelines need, and where should it go?

Less than blanket review, concentrated where your risk actually lives, and the sampling design is the whole answer. The method: classify outputs by risk tier (new-product copy in regulated-adjacent categories at the top; refreshed descriptions of stable catalog items at the bottom), then set per-tier sampling rates from consequences rather than anxiety: full review where a claim error has legal or credibility cost, statistical sampling (enough to bound error rates with confidence) for the middle, spot sampling for the benign tail. Route by signal too: items the judge scores near threshold, items with detected claims, and items from recently changed prompts all earn review priority over random draws. The arithmetic for a typical Portland brand catalog: full coverage where it matters plus statistically honest sampling elsewhere usually lands under a fifth of the review hours blanket reading would cost, with better protection, because attention concentrates where errors are expensive instead of diffusing evenly across thousands of low-stakes refreshes. The design gets revisited quarterly against what sampling actually caught.