Most teams should use the OpenAI API. A minority should not.
The reasons to self-host are specific and checkable. If you don't hit any of those triggers, the operational overhead of self-hosting isn't worth it — you're spending engineering time managing GPU infrastructure instead of building product.
Here's how to evaluate honestly.
Why OpenAI (and similar APIs) make sense for most teams
Zero infrastructure. No servers, no GPUs, no CUDA driver headaches, no scaling logic, no alerting for GPU memory errors. You make an API call, you get a response. The operational cost is a monitoring dashboard and the credit card charge.
Access to the best models, immediately. When GPT-5, Claude Sonnet 3.7, or Gemini 2.5 Pro ships, you can use it the next day. Self-hosted models lag the frontier by months to years. If model quality matters to your use case — and for most customer-facing applications it does — you're comparing the best available model against one that's a generation behind.
No ops burden. Your engineers build product, not infrastructure. The API handles availability, autoscaling, load balancing, and model serving. The GPU cluster that powers GPT-5 would cost hundreds of millions of dollars to self-host. You access it for fractions of a cent per query.
Fastest path to production. An integration with the OpenAI API can be production-ready in days. Self-hosting, tested and production-hardened, takes weeks.
Reasonable pricing at moderate scale. For most early-to-mid-stage applications, API pricing is not the bottleneck. GPT-5 mini is $0.15 per 1M input tokens. At 100,000 requests per day with an average of 1,000 input tokens each, that's $15/day — $450/month. That's not expensive enough to justify self-hosting infrastructure.
The costs and limits of API dependency
There are real downsides, and they're worth being honest about.
Your data leaves your servers. Every prompt, every document you send for analysis, every conversation turn goes to OpenAI's (or Anthropic's, or Google's) infrastructure. Both OpenAI and Anthropic offer enterprise agreements with data processing guarantees, but if your legal or compliance team has hard requirements about data residency or data handling, API-only isn't the answer.
Cost at scale. API pricing that's fine at low volume becomes meaningful at high volume. If you're making 10 million API calls per month with large context windows, you may be spending $50,000–$200,000/month. At that point, the economics of owning GPU infrastructure start to look different.
Rate limits. OpenAI enforces rate limits by tier. If your application has bursty traffic patterns or you're trying to process large batches, you'll hit limits and need to implement backoff logic, queuing, or request a limit increase. This is manageable but adds engineering complexity.
Model changes you don't control. OpenAI retires models. Behavior changes across model versions. If you've tuned your prompts carefully for GPT-5 and the model is deprecated in favor of a successor, you're re-tuning. This happens every 12–24 months on average.
Self-hosting options
If you decide self-hosting is warranted, three main approaches:
Ollama. The simplest path to running open-source models locally or on a server. Designed for ease of use. Runs Llama 4, Mistral, Phi-3, Gemma, and dozens of others with a one-line command. Good for development, internal tools, and moderate-traffic production use cases where the team doesn't want to manage complex serving infrastructure.
vLLM. An open-source LLM inference engine optimized for throughput in production. Handles continuous batching, PagedAttention for efficient GPU memory use, and high-concurrency workloads. The right choice when you need production-grade serving with real traffic. Requires more setup than Ollama but performs significantly better under load.
llama.cpp. A C++ implementation that runs LLMs efficiently on CPU (with optional GPU acceleration). No CUDA requirement. Runs on Mac hardware with Metal acceleration. Useful when you have no GPU budget, are running models on edge hardware, or want to prototype without cloud GPU costs.
Typical production setup: a single A100 80GB GPU (available on AWS as a p4d.xlarge portion, or via Lambda Labs, CoreWeave, or RunPod) running vLLM serving Llama 4 or Mistral 7B can handle moderate production traffic.
Self-hosting infrastructure costs
The math depends on your traffic, but here's a realistic baseline:
Single A100 80GB GPU:
- AWS EC2
p4d.24xlarge(8× A100): ~$32/hour; single GPU equivalent ~$4/hour - Lambda Labs A100 80GB: ~$1.89/hour
- RunPod A100 SXM: ~$2.49/hour
- CoreWeave A100 80GB: ~$2.21/hour
For a 24/7 production deployment:
- Reserved Lambda Labs A100: ~$1,360/month
- On-demand with autoscaling: ~$2,000–$4,000/month (allows scale-down during low traffic)
- Managed equivalent (AWS EC2 reserved): ~$3,500–$5,000/month
Add operational overhead:
- Engineering time to set up and maintain: 5–10 hours/week (ongoing)
- Monitoring infrastructure: $50–$200/month
- Storage for model weights: ~$20–$50/month for a 70B model
Total realistic self-hosting cost: $1,500–$5,000/month depending on GPU tier, cloud provider, and whether you run 24/7 or autoscale.
The three triggers for self-hosting
Trigger 1: Compliance requirements
HIPAA, SOC 2 Type II, FedRAMP, GDPR with strict data residency — any of these may require that data never leaves your infrastructure. OpenAI and Anthropic do offer compliant options (OpenAI Enterprise with a BAA, for example), but the coverage, contractual terms, and audit trail may not satisfy your legal team.
If your compliance counsel says "data cannot leave our environment," self-hosting is the answer.
Trigger 2: API spend exceeds $3,000/month
This is a rough threshold, not a rule. But once you're spending more than $3,000/month on LLM API calls, it's worth modeling whether self-hosting a smaller open-source model — Llama 4, Mistral, or Phi-3 — could handle the load at lower cost.
The comparison is always quality-adjusted. If GPT-5 at $3,000/month produces output quality that Llama 4 at $1,500/month can't match for your use case, the $1,500 savings doesn't justify the quality loss. But if a smaller model is good enough for your task, the economics shift significantly above certain volume thresholds.
Trigger 3: Proprietary training data that cannot leave your network
If your primary value comes from a fine-tuned model trained on data that is too sensitive or competitively sensitive to send through an external API — proprietary process data, confidential client records, internal pricing and deal structures — self-hosting may be necessary.
Note: most teams overestimate how often this applies. Fine-tuning is a different operation from inference. You can fine-tune a model on-premises and still serve it via an external API in some architectures. But if even inference calls involve proprietary context, self-hosting is warranted.
Models worth knowing for self-hosting
Llama 4 (Meta). The strongest open-source general-purpose model at this weight class. Competitive with GPT-3.5 Turbo and close to GPT-5 mini on many benchmarks. Requires at least two A100 40GB GPUs or one A100 80GB for 4-bit quantized serving.
Mistral 7B / Mixtral 8x7B (Mistral AI). Efficient and strong for their size. Mistral 7B runs on a single consumer GPU (RTX 3090). Mixtral 8x7B (a mixture-of-experts model) punches above its weight on reasoning tasks.
Phi-3 (Microsoft). Small, efficient models from Microsoft that perform remarkably well for their size on coding and reasoning tasks. Phi-3 Mini (3.8B parameters) runs on edge hardware and performs comparably to GPT-3.5 on many standard benchmarks.
The bottom line
Use the OpenAI API (or Anthropic, or Google) unless:
- Your compliance requirements explicitly prohibit data leaving your environment
- You're spending more than $3,000/month on API calls and a quality-matched open-source model exists
- Your inference calls involve proprietary data that cannot leave your network
For everything else, the operational overhead of self-hosting — GPU management, model updates, scaling logic, hardware failures — is not worth the cost savings at moderate scale. Build product, not infrastructure.
When self-hosting is warranted, start with Ollama for development and vLLM for production. Use Lambda Labs or RunPod for GPU access before committing to long-term cloud contracts.