Loading…
Hire an AI Consultant · San Francisco, CA
San Francisco companies are technically sophisticated. Most have engineers who have read the papers, experimented with the APIs, and shipped at least one AI feature. The strategic mistakes they make are not about knowing too little. They are about skipping the evaluation steps that would have caught a bad assumption early.
A team at a SaaS company spins up an engineering effort before running a 2-week feasibility test. Six months and $500k later, the problem is harder than the early experiments suggested. An independent assessment would have caught this early. The difference is whether you find the hard part before committing the headcount.
Fixed scope, fixed price, agreed before work starts. We produce written deliverables you can act on with or without us.
Tell us about your AI architecture or strategy question.
These are patterns from companies that knew the technology well. Knowing the technology is not the same as knowing whether it will work for your specific problem.
The early demo works. The engineering team is confident. The product roadmap already has the AI feature scheduled for Q3. The feasibility assessment gets skipped because the result seems obvious. At month four of the build, the production data turns out to have a distribution problem that makes the approach three times harder than the demo suggested. A 2-week assessment before committing the engineering headcount is the cheapest risk management in the project budget.
GPT-5 scores highest on most benchmarks. It is not always the right model for a specific production task. A smaller fine-tuned model trained on 2,000 examples from your domain may outperform GPT-5 on your task while costing one-tenth the inference price. Model selection requires evaluation against your data and your metric, not against a published leaderboard. We build the eval set and run the comparison.
Most AI projects that fail in production fail because of data, not because of the model. The training or evaluation data is noisier than expected, covers fewer edge cases than production traffic, or was labeled with a definition that differs from what the model needs to learn. A data quality audit before the build starts costs 3 to 5 days of analysis and surfaces these problems when they can be fixed cheaply.
Companies deploying LLMs in production without an eval suite do not know when a model provider update breaks their product. OpenAI has updated GPT-4 at least four times since the initial release. Each update changed behavior in ways that broke some downstream applications. An eval harness is a CI check that runs your golden test set on every deployment and blocks releases that regress below a threshold. Building it takes one to two engineering weeks and is not optional for a production-grade AI feature.
A structured 2-week process that answers the question: is this use case technically feasible at the quality bar required for the product, at a cost that makes the build worthwhile?
01
Translate a product requirement into a precisely defined ML task with a measurable success criterion. Vague requirements produce misleading assessments.
02
Evaluate volume, label quality, and distribution coverage. Identify gaps and estimate the cost to close them before any model work starts.
03
Test 2 to 3 model configurations against 200 to 500 real examples from your data. Measure the metric that matters for the product decision.
04
Go or no-go, with the evidence. If go: model recommendation, cost projection, and implementation approach. If no-go: what would need to change.
Four criteria. Volume: 500 to 1,000 labeled examples minimum for fine-tuning; 10 to 50 for few-shot prompting. Label quality: inter-annotator agreement above 0.75 F1 for classification tasks. Distribution: training data must include the edge cases that appear in production traffic, not just the common cases. Labeling definition: labels must match the production definition of correct, not a simplified proxy. Data that meets all four criteria produces models that generalize. Data that fails on any one usually doesn't.
Three components: a golden set of 100 to 200 labeled input/output pairs stored in version control, a test runner that measures performance against the golden set on every deployment, and a CI gate that blocks releases scoring below a defined threshold. The golden set is the most important component. It should be reviewed by a domain expert, not assembled from model outputs. Building it requires 1 to 2 weeks of engineering and annotation time. Maintaining it requires adding 5 to 10 examples per quarter as the production distribution shifts.
We have no commercial relationship with OpenAI, Anthropic, or any model provider. Our model recommendations are based on evaluation results against your data, not on referral agreements. If the right answer for your use case is an open-weight model, that's the recommendation you get.
Discovery call, written scope document, fixed price. The scope is agreed before work starts. Scope changes require a change order. No hourly billing, no retainer, no surprise invoice at the end of the month.
Written analysis, not slides. The output of a technical assessment is a document your engineering team can read and act on without a 2-hour walkthrough. We write for engineers, not for executive summaries.
Describe the use case and what you know about the data. We reply within one business day with a rough scope and price range. No commitment required.