This checklist is for hospital IT leaders, digital health founders and compliance officers scoping an AI deployment that will touch protected health information. The first time a senior clinician asks "where exactly did that patient's data go when the model produced this answer?", the deployment either has a forensically defensible answer or it does not. There is no middle ground. We have shipped healthcare AI across 30+ facilities (clinical chatbots, ambient scribing, medical RAG agents at 98.4% citation accuracy in production), and every one of them passes the same 12 controls before a clinician sees it. The list is short, boring and load-bearing.
This is the checklist we run on the first kickoff call of every healthcare engagement. If a control on the list is not in scope, we reshape the engagement or decline it. HIPAA does not negotiate, and the cost of skipping any one of these is rebuilding the entire deployment under audit pressure six months in. Better to design for it in week one.
Before any PHI moves: contract and perimeter
1. BAA signed before any PHI touches the engagement
Step zero. A signed Business Associate Agreement covers Aiinfox and any sub-processor (cloud provider, model vendor, vector store) that will touch protected health information. We will not look at a sample dataset, run a discovery call against real records, or accept screenshots that contain PHI before the BAA is countersigned. If the vendor stack you are evaluating includes any provider that will not sign a BAA (some hosted LLM providers explicitly refuse), that provider is out of scope for HIPAA work and the architecture pivots to self-hosted before any other decision is made.
2. Self-hosted model inference inside the customer VPC
Patient data should not leave the customer's network during inference. For HIPAA-scoped engagements we deploy an open-weight Llama model (70B or 8B class) on vLLM inside the customer's AWS, Azure or GCP VPC, or on-premises on customer-owned hardware for the highest-sensitivity engagements. Hosted LLM APIs are used only when (a) a BAA is in place with the provider, (b) the engagement risk tolerance accepts the egress, and (c) the data classification has been reviewed by the customer's privacy officer. The default is self-hosted. The exception is documented.
3. Vector store and retrieval index inside the perimeter
Self-hosting the LLM but using a managed vector database outside the customer cloud still leaks. The chunked, embedded representation of PHI is itself PHI for HIPAA purposes, and the retrieval-time query carries patient identifiers. Standard pattern: pgvector inside the existing customer Postgres, or Qdrant deployed on the customer's Kubernetes. Embedding models run locally on a CPU or small GPU instance. No data crosses out of the VPC during ingestion, indexing or query time. This is the perimeter rule behind every RAG deployment we ship in healthcare.
Access, redaction and the audit trail
4. Identity, access and least-privilege role design
Every user, service and model endpoint authenticates against the customer's identity provider (Okta, Microsoft Entra ID, AWS IAM Identity Center). Application roles are scoped to the smallest data set each role legitimately needs: a triage chatbot sees demographic and reason-for-visit fields, not the full chart. Service-account credentials rotate on a 30-day cadence at minimum. No long-lived tokens, no shared service accounts, no hardcoded keys in the model's prompt or tool definitions.
5. PII and PHI redaction at every untrusted boundary
Any LLM call that crosses a less-trusted boundary (hosted model API, third-party analytics, log shipping to an external observability vendor) goes through a redaction layer that strips PHI before transmission. The redaction layer is rule-based for high-confidence identifiers (MRN format, SSN, date of birth, names matched against the patient table) and supplemented by a smaller classifier for free-text identifiers. The redacted version is what crosses the boundary; the original, encrypted at rest with a customer-managed KMS key, stays inside the VPC.
6. End-to-end audit logging on every model and tool call
When a clinician three weeks from now asks why the agent produced a specific answer, the audit log must reconstruct the full chain: input prompt, retrieved context (with document IDs and citation links), model version, prompt version and temperature, tool calls with their arguments and responses, the final output, and the user identity that triggered it. The log is tamper-evident (append-only, with cryptographic chaining where the customer's compliance posture requires it), and retention matches the customer's medical-records retention policy, typically six to ten years in the US and jurisdiction-dependent elsewhere.
Clinical safety controls
7. Refusal layer for safety-critical query categories
A confidently wrong AI answer in healthcare can kill people. Every clinical AI deployment ships with an explicit refusal layer scoped to safety-critical query categories: drug dosage, drug interactions, contraindications, triage severity, dangerous procedures. On those categories the model is configured to say "I cannot answer this; escalating to a clinician" rather than guess, even when retrieval returned a partially relevant chunk. The refusal threshold is tuned against a clinician-reviewed eval set, not against a generic confidence score the model emits.
Refusal is a feature, not a bug. Our medical-inquiry RAG agent refuses roughly 6% of incoming queries. Clinicians find that 6% acceptable. They would not find a 6% hallucination rate acceptable, and that is the trade we explicitly design for in production.
8. Required citations on every clinical answer
Every retrieval-augmented answer links every clinical claim to a retrieved source: formulary entry, clinical guideline section, protocol document or prior visit note. Answers without citations are rejected before display. The citation is rendered inline with the answer, not as a footer the user can ignore, and clicking it surfaces the source passage in context. This is the structural defense against hallucination: the model is constrained by what retrieval surfaced, and the user can verify any individual claim.
9. Clinician-reviewed eval set in week one
The eval set is built before the model is selected, before the retrieval architecture is finalized, and before any user sees the system. A senior clinician on the customer side reviews 200-500 representative queries, agrees the correct answer and the correct refusal cases, and signs off on the acceptance thresholds for the safety-critical categories. This becomes the contract for the build. Every prompt change, every model swap, every chunking tweak runs against the eval set before merging. Regression past threshold blocks deployment.
10. Production drift monitoring against the eval set
CI evals catch known regressions; production evals catch unknown ones. Sample 5-10% of production traffic, run it through the clinician-validated rubric (LLM-as-judge plus deterministic checks on citation correctness, refusal correctness and retrieval recall), and alert when category-level metrics shift more than two standard deviations from baseline. Drug-dosage and triage categories get tighter thresholds and alert on a single-standard-deviation shift. The customer's clinical informatics team receives the drift report weekly.
Keys, runbooks and incident response
11. Encryption, key management and customer-managed keys
Data at rest is encrypted with customer-managed KMS keys, not the cloud provider's default keys. The customer retains control of key rotation, key access policies and the ability to revoke. Data in transit is TLS 1.2 or higher end-to-end, including between internal services; no plaintext on the internal network. Database backups are encrypted with the same customer-managed keys. The customer can audit which services have decrypt access at any point in time.
12. Documented runbooks, on-call and incident response
The HIPAA Security Rule requires documented incident response. We ship every healthcare engagement with runbooks covering common operational scenarios (model degradation, retrieval failure, refusal-rate spike, latency regression), an on-call rotation that names the responsible engineer for each business-hours window, and a breach-notification runbook aligned to HIPAA's 60-day outer limit and the customer's compliance team. If the customer's existing incident-response plan covers AI systems, we slot into it; if not, we draft the AI-specific addendum during the engagement.
Three deal-breakers worth calling out
Three things will end the engagement before the rest of the checklist matters:
- Sending PHI to a hosted LLM provider that will not sign a BAA. Non-negotiable. The architecture pivots to self-hosted Llama or another open-weight model.
- Designing an AI that takes irreversible clinical action without a human in the loop. Every Aiinfox healthcare deployment defers final clinical decisions to a clinician.
- Logging PHI to a generic observability vendor (Datadog, Sentry) without redaction. Logs are PHI when they contain identifiers. Redact at ingress to the logging layer, not at the dashboard.
Wrapping up
HIPAA-compliant AI is mostly a discipline problem, not a technology problem. Pick the controls in week one, design the architecture around them, and gate every release against the clinician-reviewed eval set. The teams we have seen succeed treat compliance as a first-class engineering requirement; the teams that retrofit it after a working demo end up rebuilding under audit pressure. Run the checklist before the build. Build for the checklist. Ship the AI clinicians actually adopt.
To scope a HIPAA-aligned AI build for your facility, whether a clinical RAG agent, ambient scribing or an AI hospital management system, start with our healthcare AI development page or book a 30-minute discovery call.
Frequently asked questions
Do we need a BAA with our LLM provider for a HIPAA-compliant deployment?
Yes, if any PHI will cross to the provider during inference. Several major hosted providers and the major clouds will sign BAAs for eligible enterprise or API tiers, but coverage varies by service and tier, so confirm it in writing before any PHI moves. Where a BAA cannot be put in place, the architecture pivots to a self-hosted open-weight model inside the customer's VPC.
Can we use ChatGPT or Claude directly in a clinical workflow?
Only if a BAA is in place with the provider, the engagement risk tolerance accepts external inference, and PHI redaction is wired in at the boundary. For most clinical workflows we recommend a self-hosted open-weight fine-tune inside the customer VPC instead: the egress concern disappears and the model can be tuned on de-identified clinical data without a third-party processor.
How long does a HIPAA-compliant AI deployment take to ship?
Six to eight weeks for a clinical chatbot or RAG agent pilot in a single facility, eight to twelve weeks for an AI HMS implementation including HL7/FHIR migration and clinician training, and around twelve weeks for a fine-tuned healthcare LLM with a curated training set and self-hosted deployment. Multi-facility rollouts replicate per site after the first, typically four weeks each.
Will the AI replace clinicians in your healthcare deployments?
No. Every Aiinfox healthcare deployment is built so the AI assists, scribes, summarizes, retrieves and flags, while every clinical decision remains with the clinician. The system defers explicitly on safety-critical queries through the refusal layer.