Running an LLM on patient data is not inherently a HIPAA violation. Running one without the right controls is. The distinction matters because a lot of healthcare organizations either avoid AI entirely because they assume it's non-compliant, or deploy it carelessly because they assume their cloud provider handles compliance automatically.
Neither assumption is correct. This is a practical guide to what HIPAA actually requires for AI systems, which deployment options meet those requirements, and what the total cost looks like.
What HIPAA requires for AI systems
HIPAA's Security Rule under 45 CFR §164.312 specifies technical safeguards for systems that touch Protected Health Information (PHI). For an AI system, the relevant requirements are:
Access controls (§164.312(a)(1)): Each user or system component that accesses PHI must have a unique identifier. Shared API keys that multiple services use are not compliant. Role-based access controls are required, and access must be terminated when a user or process no longer needs it.
Audit controls (§164.312(b)): The system must record who accessed what PHI, when, and what action was taken. For an LLM this means: logging which queries included PHI, what was retrieved, and what was returned. Logs must be retained (the retention period isn't specified in the Security Rule, but the Breach Notification Rule creates practical requirements for multi-year retention) and protected from tampering.
Integrity controls (§164.312(c)): PHI must not be altered or destroyed improperly. For inference workloads this is primarily about ensuring the model's outputs are logged accurately and responses can't be retroactively modified.
Transmission security (§164.312(e)(1)): PHI transmitted over networks must be encrypted. TLS 1.2 minimum; TLS 1.3 is preferred. This applies to the connection between your application and the LLM inference endpoint.
Business Associate Agreement (BAA): Any vendor who processes PHI on your behalf must sign a BAA. This is the most commonly misunderstood HIPAA requirement in AI deployments. Your cloud provider handling PHI must sign a BAA with you. Your LLM provider (if you're calling an external API) must sign one too.
Which providers can sign a BAA
This is the most practically important question and the one where AI teams most often get it wrong.
Microsoft Azure OpenAI Service: Can sign a BAA. Azure's HIPAA compliance covers the OpenAI models hosted on Azure infrastructure. This means you can use GPT-5, GPT-4, and other models available in Azure OpenAI under a BAA arrangement, as long as you configure the service within a compliant Azure environment. Azure has a standard HIPAA BAA available to all customers in their Online Service Terms.
Google Vertex AI (including Gemini models): Can sign a BAA. Google's Business Associate Amendment is available for Google Cloud services including Vertex AI. Gemini 2.5 Pro and other models available through Vertex AI are covered. You need to configure a Google Cloud organization with HIPAA controls and sign the amendment.
OpenAI API (api.openai.com): Cannot sign a standard BAA for the standard API. OpenAI's terms of service explicitly state that the API is not a HIPAA-covered service and they do not currently enter into BAAs for standard API access. If you are sending PHI to api.openai.com, you are likely in violation of HIPAA.
AWS Bedrock: AWS can sign a BAA covering Bedrock. Models available through Bedrock (Claude, Llama, Titan, Mistral, and others) are hosted on AWS infrastructure. AWS's HIPAA BAA is available and covers Bedrock for eligible workloads.
Anthropic API (api.anthropic.com): Anthropic now offers BAAs for eligible API customers, typically alongside zero-data-retention terms — verify current availability and scope with Anthropic directly, as terms change. Claude is also available through AWS Bedrock and Google Vertex AI, both of which support BAAs, if you prefer to route HIPAA workloads through a cloud provider you already have a BAA with.
Deployment option 1: Azure OpenAI with BAA
The fastest path to a compliant LLM for most healthcare organizations.
Setup: Deploy Azure OpenAI Service in an Azure subscription. Enable Azure Private Endpoints so inference traffic stays on the Azure backbone and never traverses the public internet. Configure Azure Active Directory authentication — no shared API keys. Enable Azure Monitor and Diagnostic Settings to log all API calls including the request content (necessary for audit trails).
Compliance posture: Azure handles the infrastructure-level controls. You are responsible for: ensuring PHI isn't included in model training feedback (disable the "allow training" setting, which Azure exposes), implementing application-level access controls, and maintaining audit logs.
Limitations: You're restricted to models Microsoft has deployed to Azure OpenAI. Not every model version is available. Regional availability affects which models you can use.
Cost: Azure OpenAI pricing is comparable to direct OpenAI API pricing — verify current per-token rates before budgeting. For a clinical documentation assistant handling 100 notes per day at ~3,000 tokens each on a frontier model like GPT-5, expect roughly $300–$900/month in inference costs alone.
Deployment option 2: Self-hosted on AWS VPC
Higher infrastructure cost, full control over the model and data path.
Stack: vLLM for inference serving, an open-weight model such as Llama 4 (a smaller or larger variant depending on quality requirements), pgvector for any retrieval components, all running within an AWS VPC with no public internet exposure.
Infrastructure:
- A small open model (~8B class, e.g. Llama 4 Scout): runs on a single g5.2xlarge ($1.00/hr) or p3.2xlarge ($3.06/hr). Handles ~50–100 concurrent requests depending on context length.
- A larger open model (~70B class): requires a g5.48xlarge ($16.29/hr) or two A10G instances behind a load balancer. 4-bit quantized (GPTQ or AWQ) brings this down to ~$6–8/hr on a g5.12xlarge.
- Add an Application Load Balancer, RDS for PostgreSQL with pgvector, and CloudWatch for logging.
Audit logging: Configure vLLM to log all requests and responses. Store these logs in CloudWatch Logs, then stream to S3 with Object Lock enabled (WORM storage) for tamper-evident retention. Apply KMS encryption with customer-managed keys.
Compliance posture: You control the entire data path. No PHI leaves your VPC. BAA with AWS covers the infrastructure. This approach also works for organizations with data residency requirements (EU, certain state-level requirements).
Cost: Expect $3,000–$8,000/month for a production-ready self-hosted setup covering inference, storage, networking, and backup. This is higher than managed API costs at low volume but becomes cost-competitive above ~500,000 tokens/day.
Operational requirement: Someone on your team or your vendor must maintain the infrastructure. Model updates aren't automatic. Security patches for vLLM, the OS, and dependencies require active management.
Deployment option 3: Google Vertex AI with Gemini
Similar to Azure OpenAI in structure but with different model availability.
Setup: Google Cloud project with HIPAA controls configured (VPC Service Controls, Cloud Audit Logs, Customer-Managed Encryption Keys). Sign Google's BAA via Google Cloud console under Account → Legal → Health Data Processing Amendment.
Advantages: Gemini 2.5 Pro's 1M token context window is valuable for long clinical documents. Google's document processing capabilities integrate well with Vertex AI.
Limitations: VPC Service Controls configuration is more complex than Azure's Private Endpoints. The setup requires more Google Cloud expertise.
Audit logging requirements
HIPAA doesn't specify a log format, but an audit trail must answer: who, what, when.
For an LLM deployment, implement logging at two levels:
Application level: Log the user ID (or system ID), timestamp, query hash or summary (not necessarily the full PHI text, which would create its own storage/retention challenges), and whether PHI was included based on your detection logic. Log the response summary and which retrieved documents were used (for RAG systems).
Infrastructure level: CloudWatch (AWS) or Azure Monitor logs capture API calls, authentication events, and compute-level activity. Enable these with at minimum 6-year retention (to cover HIPAA's 6-year record retention requirement).
Tamper protection: Write audit logs to a separate storage account or S3 bucket with Object Lock / WORM enabled. The logging system should have write-only access from the application — the application can write logs but cannot read or modify them.
What this actually costs
Realistic total cost ranges for a HIPAA-compliant LLM deployment in production:
| Approach | Setup cost | Monthly ongoing | Best for | |---|---|---|---| | Azure OpenAI with BAA | $5,000–$15,000 | $500–$3,000 | Fast start, moderate volume | | AWS Bedrock (Claude/Llama) | $8,000–$20,000 | $1,000–$5,000 | AWS-native orgs, flexible model choice | | Self-hosted vLLM on AWS | $15,000–$40,000 | $3,000–$10,000 | High volume, data sovereignty requirements | | Google Vertex AI | $8,000–$20,000 | $800–$4,000 | Long-context use cases, GCP orgs |
Setup costs include infrastructure configuration, security controls, audit logging implementation, and BAA execution. They do not include application development (the clinical feature built on top of the LLM).
What people get wrong
Assuming the cloud provider's HIPAA certification covers the AI layer. AWS, Azure, and GCP are HIPAA-compliant infrastructure providers. That means the data centers, networking, and storage meet the requirements. It does not automatically mean your application, your prompt design, or your logging setup is compliant. The Shared Responsibility Model applies to compliance the same way it applies to security.
Sending PHI to the standard OpenAI API. This happens more than it should. If your application passes patient records, clinical notes, or any identifying information to api.openai.com, you likely have a HIPAA problem. Audit your API calls before assuming you're in the clear.
Not logging the full request and response path. Some teams log that an LLM call was made but not what was asked or returned. For a HIPAA audit, this is insufficient. You need to be able to reconstruct what PHI was processed and what the system did with it.
No PHI detection layer. Even if your application isn't supposed to include PHI in prompts, users will inevitably paste it in. Implement a PHI detection step (AWS Comprehend Medical, Google Healthcare NLP, or a fine-tuned classifier) that flags or redacts PHI before it reaches the model endpoint.
The bottom line
A HIPAA-compliant LLM deployment is achievable with existing infrastructure from Azure, AWS, and Google. The non-negotiables are: a BAA with every vendor who touches PHI, access controls with individual identifiers, tamper-evident audit logging, and encryption in transit and at rest.
The biggest risk is not the technology — it's the assumption that compliance is handled by someone else. Map every component in your data path, confirm which ones have signed BAAs, and verify that your audit logging covers who accessed what PHI and when.
If you're building a healthcare AI feature and want to understand the compliance architecture before committing to a deployment approach, see our HIPAA-compliant AI services page for how we structure these engagements.