Loading…
AI Product Manager: San Francisco
SaaS companies in San Francisco add AI features under investor and customer pressure. The product team writes specs. The problem is that AI feature specs require four things standard specs do not: evaluation criteria, a test dataset, failure mode handling, and a model versioning policy.
A spec that says "the feature summarizes documents" is not a spec. A spec that says "the feature produces summaries that pass a 5-point rubric at 80% pass rate on 50 test documents, with latency under 4 seconds at p95" is a spec. The difference between these two documents is the difference between a feature that ships and one that engineers argue about for three months.
We provide that product function for a fixed engagement over 8–12 weeks.
Tell us about your AI feature spec problem.
A regular product spec defines what the feature does, for whom, and what the acceptance criteria are. An AI feature spec must do the same and then answer three questions the standard spec ignores.
What does "working correctly" mean in measurable terms? An AI feature has a probability distribution of outputs, not a deterministic result. The spec must define the rubric and the pass rate that constitutes acceptable quality. Without this, engineering can never declare the feature done.
What happens when the model is wrong? Every AI feature fails sometimes. The spec must define whether the user is informed, what confidence threshold triggers a fallback, and whether there is a human review path for low-confidence outputs. Figma, Notion, and other SF-based product companies that ship AI features well have this in every spec.
Who owns the feature when the model provider updates? OpenAI releases a new GPT-5 version. Anthropic releases Claude 4. The spec must include a model versioning policy that answers this question before the update happens.
Two requirements that separate shippable AI feature specs from feature ideas. Both need to be in the spec before engineering writes a line of code.
Every AI feature spec includes: a rubric with 3–7 criteria the output must meet, a pass threshold (e.g., 80% of test outputs pass all rubric criteria), a test dataset of at least 50 labeled examples, and a measurement protocol specifying who scores the outputs and how disagreements are resolved. These four elements define done.
When a provider releases a new version, someone must evaluate it against the rubric, make the migration decision, and document the result. The policy specifies: who owns this evaluation, how long it takes, what criteria trigger migration, and what the rollback plan is. Without a policy, migrations happen silently and quality changes go unattributed.
What does the AI feature do when its confidence is below the threshold? Display the output with a caveat? Fall back to a rule-based result? Route to a human reviewer? The spec defines this explicitly. Features that ship without a failure mode spec handle low-confidence outputs inconsistently across the codebase.
Each AI feature recurs this question. The decision framework asks: does a vendor API pass the evaluation rubric at acceptable cost? Does the feature need proprietary data? What is the total cost of ownership for each path? We document the answer for each feature before recommending an approach.
Click-through rate measures whether users clicked on the output. It does not measure whether the output was correct. For a document summarizer, a user might click on a summary that misrepresents the document. The click metric would show the feature working. The feature would be failing.
AI feature experiments require human evaluation. A sample of outputs from each variant is presented to raters using a blind protocol. Raters do not know which variant produced which output. Raters score each output against the rubric. Inter-annotator agreement, the percentage of cases where two raters agree, validates that the rubric is interpretable before you trust the experiment results.
This design requires a PM who understands evaluation methodology. At companies like Scale AI and Cohere, this is standard practice. At most SaaS companies adding their second or third AI feature, it is not, and A/B test results are meaningless as a result.
Discovery call (1 hour)
We review your current AI feature backlog, your existing specs, and your evaluation setup. We identify which features are spec-complete and which need evaluation criteria written. We quote a fixed price at the end of this call.
Spec sprint (weeks 1–6)
We write evaluation criteria, test datasets, failure mode specifications, and model versioning policies for each feature in scope. We review with engineering and update based on technical constraints.
Build support and handoff (weeks 7–12)
We support engineering through the build, run evaluation against the test dataset, and document the experimental design for the first A/B test. The engagement ends with specs your team can execute independently.
Tell us about your AI feature backlog and the specs you have written so far. We'll reply within one business day with a rough scope and price range.