LLM cost optimization in New York.

The technical reality of LLM spend in New York workloads.

Questions from New York buyers.

What savings are realistic, and where do they usually come from?

For workloads that have never been cost-engineered, 40 to 70 percent reductions are common; for already-tuned systems, 10 to 25 percent is the honest ceiling. The sources rank consistently. Prompt bloat first: system prompts and context that grew during development and never got trimmed, often 30 to 60 percent of tokens doing nothing. Model oversizing second: frontier models handling classification and extraction a model a tenth the price does identically, provable on your eval set. Missing caching third: repeated context (policy documents, templates, filing boilerplate) re-sent and re-billed every call when prompt caching would cut it sharply. Synchronous-where-batch-fits fourth: overnight document runs paying real-time prices when batch APIs cost half. We baseline a month of actual usage before promising anything, and the audit report ranks fixes by dollars, not elegance.

Our research and filings workloads re-read the same documents constantly. How much does caching actually recover?

On repeated-context workloads, prompt caching is usually the single biggest line item. The mechanics: when consecutive requests share a long prefix (a credit agreement under analysis, a 10-K being queried, standing compliance instructions), cached input tokens bill at a steep discount (90 percent off on Anthropic's current pricing) and time-to-first-token drops too. A research workflow where analysts run twenty queries against the same filing pays full price once and the discount nineteen times, if the prompt architecture puts the document in a stable prefix, which is precisely the engineering. We restructure prompts for cache stability (stable system content first, volatile queries last), measure hit rates in production, and typically see 50 to 80 percent input-cost reductions on document-analysis workloads shaped like New York's.

Can we route to cheaper models without quality drops our compliance and legal reviewers would notice?

Yes, if routing is gated by your eval set rather than by optimism. The method: build a labeled evaluation set from your real workload (the same one your reviewers would grade), benchmark candidate models on it, and route only the request classes where the cheaper model matches the incumbent within your tolerance. In regulated New York workflows, that typically means classification, extraction, and routine summarization move to small models with no perceptible change, while final-draft generation for client-facing or filed material stays on the strong model. The router itself is observable: every request logs which model served it and why, so when a reviewer questions an output, the answer is in the log, not a shrug. Quality regressions are caught by the same eval set running on every change.

What if the audit finds our spend is already reasonable?

Then the report says exactly that, and the engagement cost was the price of knowing. It happens: teams that built with caching, right-sized models, and batch processing from the start sometimes have only single-digit percentage headroom, and implementing further optimization would cost more than it returns. In that case you receive the baseline instrumentation (which most teams lack and keep), a ranked list of the marginal items with their honest payback math, and a written 'your spend is sound' that finance can file. What we will not do is invent savings to justify the engagement or push architectural churn with a two-year payback. The audit tier exists precisely so the question is cheap to answer instead of expensive to discover later. Roughly one in five audits ends this way, and those clients tend to come back when usage has tripled and the answer has changed.