LLM cost optimization in Boston.

The technical reality of compliant-channel spend in Boston.

Questions from Boston buyers.

Can we cut costs without leaving our BAA-eligible inference path?

Yes; every major lever works inside the compliant channel. Azure OpenAI and Bedrock both offer batch processing at roughly half price, prompt caching with the same steep input discounts, and a range of model sizes under the same BAA umbrella, so routing, caching, and batch all apply without touching the compliance posture. The audit treats the BAA boundary as a constraint, not an excuse: we right-size models within the eligible lineup (the gap between a frontier model and a small model on classification is just as large inside Azure as outside), restructure prompts for cache stability, and move overnight clinical-document runs to batch. What the boundary does change is the comparison set; we benchmark within your eligible options rather than against consumer endpoints you cannot use, and the savings numbers in the report are all achievable inside your current agreements.

Is prompt caching safe with PHI, or does caching create a compliance problem?

Provider-side prompt caching is a billing and latency optimization, not data storage you manage, but the design still deserves PHI-aware care. The safe pattern: keep the cached stable prefix PHI-free (system instructions, clinical glossaries, policy text, schemas are identical across requests and cache beautifully), and let patient-specific context ride in the volatile suffix that is never reused across patients. That gets you most of the discount, since the heavy stable content is exactly what caching rewards, while patient data never becomes a shared artifact even at the billing layer. Response caching (storing outputs to skip repeat calls) is different and does live in your infrastructure: there it is standard PHI engineering, encrypted, access-controlled, and audit-logged like any other PHI store. The report documents both designs so your privacy officer signs off on architecture, not assurances.

At hospital scale, does self-hosting open-weight models beat the API bill?

Later than enthusiasm suggests, and the audit runs your actual numbers. The honest math: self-hosting carries GPU capacity (bought or rented), serving infrastructure, model updates, and the engineering attention those consume, against API pricing that has fallen steadily and includes batch and caching discounts. The crossover typically arrives when sustained, high-utilization volume (not peak, sustained) keeps expensive GPUs busy, and when the workload tolerates open-weight quality on your eval set. Many Boston clinical workloads fail the second test before the first matters: extraction and summarization pass, complex clinical reasoning often does not yet. Where self-hosting does win here, it usually wins on data-policy grounds rather than cost (research corpora that cannot leave campus infrastructure), and then the budget question is honest: you are paying for control, and the report prices that premium explicitly.

Audit logging and compliance overhead add tokens and storage. How much of the bill is that, really?

Far less than teams fear, and almost never the right place to economize. The token side is negligible: logging captures prompts and outputs that already existed; it adds storage, not model calls. Storage for immutable audit logs at clinical volume runs tens of dollars a month at S3-class pricing, rounding error against inference. The genuine compliance-adjacent costs are subtler: refusal and confidence scaffolding adds modest output tokens (worth it; that is the safety property), version-pinned models forgo newer-cheaper releases until revalidation (a real but manageable lag we schedule rather than skip), and conservative timeouts retry more than aggressive ones (tunable with data). The audit separates these from the actual cost drivers (prompt bloat, oversizing, missing caching) so compliance never takes the blame for engineering debt, and the savings plan leaves the audit trail exactly as your reviewers expect it.