LLM cost optimization in San Francisco.

The technical reality of AI COGS in SF products.

Questions from San Francisco buyers.

Our AI features are eating gross margin. What does optimization actually do to the margin line?

It converts AI COGS from a scary unknown into an engineered number, and the movement is often dramatic. The arithmetic: an AI SaaS charging $99 per seat with $30 of monthly inference per active user runs a 70-percent-feeling margin that is really much worse on power users. The typical fix stack (prompt trimming, caching, routing the easy 70 percent of requests to a model at a tenth the price, batch for non-interactive work) moves per-user inference toward $6 to $12 without measurable quality loss, because the changes are gated on your eval set. That is the difference between AI features as a margin story investors question and one you present proudly. We model the before-and-after per-user economics in the audit, so the board slide writes itself from measured data.

How does prompt caching work, and why is everyone suddenly talking about it?

Because the discount is enormous and most prompt architectures waste it. Mechanics: providers cache the processed state of a prompt's prefix; when the next request shares that exact prefix, cached tokens bill at a fraction of price (90 percent off on Anthropic's current rates) and latency drops. The catch is exactness: caches key on identical prefixes, so a prompt that interleaves stable content (system instructions, schemas, knowledge snippets) with volatile content (user query, timestamp, session data) never hits. The engineering is reordering: stable content first and byte-identical across requests, volatile content last, and session-specific data out of the prefix entirely. Products with heavy system prompts and tool definitions routinely see 50 to 80 percent input-cost cuts from this restructuring alone, which is why it is the first thing we check in an SF audit.

When does fine-tuning a small model to replace a frontier model actually pay off?

When three conditions hold simultaneously: the task is narrow and stable (classification, extraction, formatting, your specific summarization style, not open-ended reasoning), volume is high enough that the per-token spread pays back the project (usually north of several million tokens monthly on the task), and you have or can generate training data (often the frontier model's own validated outputs). When they hold, a fine-tuned small model matches frontier quality on that narrow task at 5 to 15 percent of the cost, and the payback math is weeks. When they do not (low volume, shifting requirements, genuinely hard reasoning), fine-tuning is a science project with a maintenance tail, and routing plus caching deliver most of the benefit without owning a model. The audit runs this math per task; most SF products have one or two genuine candidates and a temptation toward five.

Investors are asking about our AI unit economics in diligence. What should we be able to show?

Four artifacts, all of which a proper optimization pass produces as side effects. Cost per active user and per feature, trended monthly, from real metering rather than invoice division. Marginal cost per core action (per document processed, per generation) so pricing decisions have a floor under them. The routing and caching posture, demonstrating the spend is engineered, not endured; diligence increasingly distinguishes teams who can explain their token economics from teams who cannot. And quality evidence tied to cost changes: the eval set results showing the cheap path performs, which preempts the 'did you trade quality for margin' question. We deliver all four as standard, and founders tell us the diligence conversation changes tone when the answer to 'what is your inference cost' has decimal places and a trend line.