How to build a secure AI stack in digital health

By Mat Steinlin, Head of Information Security

Last updated April 2026

A Business Associate Agreement (BAA) governs what your LLM provider does with your data. It doesn't configure your infrastructure, scope your API keys by application and environment, or stop a developer from using Claude.ai to debug a production issue with a real patient chart.

Most digital health teams building AI features have done some version of the compliance work: signed a BAA with their LLM provider, set up request logging, checked the HIPAA boxes. That's real and necessary work. But it's not a security posture.

Compliance and security overlap considerably, but they're different questions. HIPAA tells you what the law requires. This guide covers what a genuinely secure AI stack actually needs, which failure modes show up most often in practice, and how to build controls that hold up when something goes wrong, not just when an auditor shows up.

If you need the compliance baseline first, start with HIPAA-Compliant AI: What Developers Need to Know. That guide covers BAA requirements, audit logging standards under 45 CFR 164.312(b), encryption, and de-identification under the Privacy Rule. This guide picks up where that one ends.

Compliance vs. security: why a BAA isn't enough

A BAA establishes a legal relationship with your LLM provider. When you sign one, the provider agrees to handle your data as a business associate under HIPAA: implementing appropriate safeguards, not using it for unauthorized purposes, and reporting breaches. When you sign a BAA with OpenAI or Anthropic, they typically configure zero data retention and opt your account out of training. That's what the BAA does.

What it doesn't do is configure anything on your side. Your key management, logging infrastructure, access controls, and developer governance are still entirely your responsibility. A developer at your company can still paste patient records into a consumer AI product with no BAA. A runaway process can still burn through your entire monthly token budget overnight. Your audit logging can still fail silently for three weeks without anyone noticing.

The NIST AI Risk Management Framework frames AI security as a multi-layered problem: the model, the system built around it, and the organization operating it. A BAA addresses one aspect of the model layer. This guide covers the rest.

The healthcare AI threat model

Before you start building controls, get the threat model right. Healthcare AI security content written for general software companies focuses on adversarial attacks: prompt injection, jailbreaking, model inversion. Those threats exist, and some matter for certain healthcare use cases. But for most digital health teams, the more pressing failure modes are organizational and infrastructural, not adversarial. Here's what actually goes wrong.

Shadow AI and unsanctioned tools

A developer uses Claude.ai to summarize clinical notes while debugging a parsing issue. No BAA covers Claude.ai for your organization. No logging captures what was sent. The developer isn't trying to create a compliance incident; they're trying to fix a bug. But the data handling is identical to a breach: PHI in a consumer product with no business associate agreement and no audit trail.

This isn't a rare edge case. It happens constantly at healthcare companies, and most discover it by accident; someone mentions a tool in standup or a security review surfaces it. The pattern is documented in incident reports and shows up in audit findings as "workforce members using unauthorized third-party tools to process PHI."

The right framing isn't "developers are reckless." It's "the approved path has friction, so people work around it." Policy without tooling doesn't work here. See Shadow AI in healthcare for the full treatment.

PHI mixing across environments and scopes

Most LLM integrations start with a single API key per provider, shared across development, staging, and production. It's convenient and becomes a liability quickly.

When a single key serves all environments, a test incident in development affects the same billing account as production. A runaway loop in dev can exhaust the org-level budget and block production requests. Any log query against that key returns a mixed-environment mess that's useless for compliance attribution. And when you need to rotate it, you break everything at once.

The blast radius problem is worse. If a key is compromised, or a misconfigured feature starts sending unexpected data, the scope of exposure is the entire organization's LLM usage — not a single application.

Audit logging gaps

The HIPAA Security Rule requires mechanisms to record and examine activity in systems that contain PHI. For LLM usage, that means logging every prompt and response that involves patient data: the content, timestamps, which model was used, and who or what initiated the request.

The logging gap that surfaces most often in practice isn't "we didn't know we needed to log." It's "we thought we were logging and we weren't." Logging infrastructure fails silently. A misconfigured log drain, a full disk, a permissions change, a deploy that broke the logging wrapper: any of these can cause a multi-week logging gap that you discover when an auditor asks for records from a specific period.

A secondary gap: teams route LLM logs to general application aggregators without accounting for what those logs contain. Because LLM audit logs include PHI in prompts and responses, they require encrypted storage and access controls separate from the rest of your log infrastructure. The audit finding that surfaces most often isn't "you didn't log," but "logs were retained without appropriate access controls."

Key sprawl and attribution loss

Over time, most teams accumulate a pile of API keys: one for a developer's experiment, another for an integration, a few more that nobody remembers creating. Nobody tracks which key belongs to which system. When a security incident occurs, or when an auditor asks "which systems were accessing patient data on a given date," the answer is "we're not sure."

Attribution isn't optional under HIPAA. 45 CFR 164.312(b) requires the ability to examine activity, which requires knowing which system generated which activity. An unattributed key makes that impossible.

When adversarial attacks do matter

Adversarial techniques become significant security concerns in specific circumstances: patient-facing interfaces where users can submit arbitrary input, agentic workflows where the LLM can take actions based on what it reads (query records, send messages, update systems), and RAG pipelines that process externally sourced or user-submitted documents.

For internal tooling where all input comes from authenticated staff and the model has no action-taking capability, the threat model is substantially lower, and many healthcare AI features are in exactly that bucket. Be honest about where yours sits before investing in adversarial mitigations.

The OWASP LLM Top 10 is the authoritative reference for this threat category. PortSwigger's web security academy covers practical attack patterns with technical depth. The healthcare-specific framing (when these matter for your use case) is in Prompt injection in healthcare AI and Agentic AI in healthcare.

The control layer: what you need before you go to production

A fully governed healthcare AI stack requires a tiered set of controls. Not all of them are equally urgent, and the common mistake is treating the full list as an undifferentiated backlog to clear before shipping anything. It's not. Some controls are required before any PHI touches the stack, some are security best practice worth getting to before significant PHI volume or a Series A, and some can wait until you have a dedicated security function.

Required before any PHI touches the AI stack:

• BAA with every LLM provider that will process PHI (covered in HIPAA-Compliant AI)

• Audit logging of PHI-touching requests with encrypted storage

• Scoped API keys: at minimum, separate keys for production vs. everything else

• Model access controls: production restricted to approved, tested models

• An approved-tools policy that developers know about and can actually follow

Security best practice (implement before significant PHI volume or Series A):

• Environment-level key isolation: separate keys for dev, staging, and production

• Security metadata in audit logs (see below) beyond the HIPAA minimum

• Cost controls per scope with hard limits that stop requests

• De-identification for features processing full clinical notes or discharge summaries

• Log drain to long-term storage with tested delivery verification

• Monitoring on logging infrastructure to detect silent failures

Defer until you have a security team:

• Automated anomaly detection on LLM usage patterns

• Penetration testing of LLM-integrated features

• Advanced per-request threat analysis

• Comprehensive developer AI tool inventory and enforcement

The rest of this guide covers the controls in the first two tiers.

Key management and scope isolation

The default LLM integration pattern is a single API key per provider. It works for getting started and fails as a governance model the moment you have more than a few developers or more than one environment.

The alternative is scoped keys: separate keys organized by application, environment, and use case. Each scope has its own key, its own model access list, and its own budget limit. A key for production summarization can't be used for development experimentation. A key for an internal tool can't access models approved only for patient-facing features. When one scope misbehaves or is compromised, the blast radius is contained to that scope.

A minimal scoped key implementation in Python, using environment variables named by scope with an explicit model allowlist per environment:

import os
from dataclasses import dataclass, field
from openai import OpenAI

@dataclass
class ScopedLLMClient:
    """
    LLM client with scope-based key isolation and model access controls.

    Each scope uses its own API key. Production environments are restricted
    to an approved model allowlist. Development environments can experiment freely.
    Attribution metadata is attached to every request for audit logging.
    """
    scope: str       # e.g. "prod-summarization", "dev-general"
    environment: str # "production" | "staging" | "development"
    application: str # e.g. "care-notes-app", "internal-tools"

    # Model allowlist per environment. None means all models are permitted.
    _ALLOWED_MODELS: dict = field(default_factory=lambda: {
        "production": {"gpt-4o", "claude-sonnet-4-6"},
        "staging":    {"gpt-4o", "gpt-4o-mini", "claude-sonnet-4-6"},
        "development": None,
    })

    def __post_init__(self):
        key_env_var = f"OPENAI_KEY_{self.scope.upper().replace('-', '_')}"
        api_key = os.environ.get(key_env_var)
        if not api_key:
            raise EnvironmentError(
                f"No API key configured for scope '{self.scope}'. "
                f"Expected environment variable: {key_env_var}"
            )
        self._client = OpenAI(api_key=api_key)
        self._allowed = self._ALLOWED_MODELS.get(self.environment)

    def complete(self, model: str, messages: list, **kwargs):
        if self._allowed is not None and model not in self._allowed:
            raise ValueError(
                f"Model '{model}' is not approved for environment '{self.environment}'. "
                f"Approved models: {self._allowed}"
            )
        return self._client.chat.completions.create(
            model=model, messages=messages, **kwargs
        )

# Usage:
# prod = ScopedLLMClient("prod-summarization", "production", "care-notes-app")
# dev  = ScopedLLMClient("dev-general", "development", "dev-tools")

In this pattern, separate environment variables hold separate API keys. A developer in the development environment uses a key that physically can't touch production budgets, production models, or production audit trails. Rotating a key for one scope doesn't affect others.

If your team uses a gateway layer (LiteLLM, Portkey, or Aptible's AI Gateway), scoping is handled at the infrastructure level and your application code passes a scope identifier or uses a pre-scoped key issued by the gateway. That approach is more robust: it centralizes key management and can enforce model controls server-side regardless of what the application code requests.

For the full treatment of key architecture, rotation patterns, and attribution requirements, see API key management and scope isolation.

Audit logging for security and compliance

The HIPAA audit control standard (45 CFR 164.312(b)) requires mechanisms to record and examine activity in systems containing PHI. For LLM interactions, that means logging prompts, responses, timestamps, and attribution for every request that handles patient data. The compliance logging implementation is in the HIPAA guide.

Security logging goes further. The fields HIPAA requires are enough to answer "did this request occur?" They're not enough to answer "why did costs spike 400% between 2am and 4am?" or "which scope was responsible for 15,000 requests in a 10-minute window?"

A logging wrapper that captures both compliance-required fields and security-useful metadata:

import json
import logging
from datetime import datetime, timezone
from openai import OpenAI

# This logger must write to encrypted, access-controlled storage.
# Do not share a log drain with general application logs — these records contain PHI.
security_logger = logging.getLogger("llm.security")

class SecurityAwareLoggingWrapper:
    """
    Logs LLM interactions with both HIPAA-required fields and security metadata.

    HIPAA-required: prompt content, response, timestamp, user attribution, model.
    Security metadata: scope, environment, request duration, token counts, status.

    Duration and token counts are anomaly signals. Unusually long requests or
    unusually large token counts can indicate prompt injection, runaway loops,
    or model misuse. Without these fields, you cannot detect or investigate them.
    """
    def __init__(self, client: OpenAI, scope: str, application: str, environment: str):
        self.client = client
        self.scope = scope
        self.application = application
        self.environment = environment

    def chat_completion(self, user_id: str, messages: list, model: str, **kwargs):
        start_time = datetime.now(timezone.utc)
        response = error = None

        try:
            response = self.client.chat.completions.create(
                model=model, messages=messages, **kwargs
            )
            status = "success"
        except Exception as e:
            status = "error"
            error = type(e).__name__
            raise
        finally:
            duration_ms = int(
                (datetime.now(timezone.utc) - start_time).total_seconds() * 1000
            )
            log_entry = {
                # HIPAA-required fields
                "timestamp":   start_time.isoformat(),
                "user_id":     user_id,
                "model":       model,
                "messages":    messages,   # Contains PHI — encrypted storage required
                # Security metadata
                "scope":       self.scope,
                "application": self.application,
                "environment": self.environment,
                "duration_ms": duration_ms,
                "status":      status,
                "error":       error,
                "input_tokens":  getattr(getattr(response, "usage", None), "prompt_tokens", None),
                "output_tokens": getattr(getattr(response, "usage", None), "completion_tokens", None),
                "request_id":    getattr(response, "id", None),
            }
            security_logger.info(json.dumps(log_entry))

        return response

The fields worth adding beyond the HIPAA minimum:

• scope and application: attribution. Without these you can't trace which system generated which requests during an incident.

• duration_ms: anomaly signal. Unusually long requests can indicate prompt injection or an overloaded upstream. A sudden increase in average duration is often the first signal of a problem.

• input_tokens and output_tokens cost signal and abuse signal. A spike in input token counts may indicate someone is sending unusually large context (intentionally or not).

• environment separates production activity from development noise in compliance logs.

One critical note on storage: these logs contain PHI. They need to be encrypted at rest, access-controlled separately from general application logs, and retained according to your retention policy (typically six years under HIPAA). Storing PHI-containing audit logs in plaintext Elasticsearch or an unencrypted S3 bucket is a compliance gap that's often worse than not logging at all, because you've created a second PHI repository you may not have accounted for.

For the full treatment (anomaly detection on log data, log drain setup, and verifying that logging is actually working), see Audit logging for healthcare AI. For HIPAA retention requirements and storage standards, see Audit log retention.

PHI handling as defense in depth

De-identification isn't required under HIPAA if you have a valid BAA with your LLM provider. The Privacy Rule's de-identification standard applies when you want to share data without PHI status, not when you've covered the BAA requirement.

That said, de-identification provides security value beyond compliance even when a BAA is in place:

• It limits what the provider's infrastructure ever sees in identifiable form.

• It limits what ends up in audit logs, which are a larger and longer-lived attack surface than individual LLM calls.

• It reduces blast radius if a provider has a security incident or a misconfigured feature starts logging more than intended.

• It creates a cleaner boundary between data the model needs and data it doesn't.

Whether this tradeoff is worth it depends on the use case. For a feature processing full clinical notes or discharge summaries where meaningful PHI reaches the model, the defense-in-depth argument is strong. For a feature doing narrow extraction on structured data fields, the added latency and implementation complexity may not justify it.

The implementation in HIPAA-Compliant AI covers the structured text case: token-based replacement of identifiers like SSNs, MRNs, and dates. Clinical notes and free-text fields require NLP-based approaches (spaCy, AWS Comprehend Medical, Microsoft Presidio) that can detect PHI in unstructured text. That's a substantially more complex problem. The deep dive with code examples for unstructured clinical text is in PHI de-identification as a security control.

Cost controls as abuse detection

LLM cost controls are typically framed as a cost management feature. But they're also a security control, and that framing changes how you design them.

A runaway cost event has two possible causes: a bug (an infinite loop, a misconfigured retry policy, a prompt that generates unexpectedly large responses) or an attack (an adversarial prompt that causes the model to generate enormous amounts of text, or a compromised key being used to run a high-volume operation). In either case, a hard limit that stops requests when a budget threshold is reached contains the damage to the scope where the problem occurred.

The architectural implication: budget limits should be per scope, not per organization. An organization-level limit means a runaway development experiment can exhaust the budget for production patient care features. Scope-level limits keep the blast radius contained.

A pre-request budget check pattern:

import httpx

class BudgetExceededError(Exception):
    pass

def check_scope_budget(scope: str, gateway_url: str, gateway_token: str) -> None:
    """
    Check remaining budget for a scope before dispatching an LLM request.
    Raises BudgetExceededError if the scope has hit its limit.

    Fail closed: if the budget endpoint is unreachable, treat it as a budget
    exceeded condition. An unavailable budget check is less costly than an
    uncontrolled spend event.
    """
    try:
        response = httpx.get(
            f"{gateway_url}/scopes/{scope}/budget",
            headers={"Authorization": f"Bearer {gateway_token}"},
            timeout=2.0,
        )
        response.raise_for_status()
        budget = response.json()
    except (httpx.HTTPError, httpx.TimeoutException) as e:
        raise BudgetExceededError(
            f"Budget check unavailable for scope '{scope}': {e}. "
            "Failing closed to prevent uncontrolled spend."
        )

    if budget["remaining_usd"] <= 0:
        raise BudgetExceededError(
            f"Scope '{scope}' has exhausted its budget. "
            f"Limit: ${budget['limit_usd']:.2f}, "
            f"Used: ${budget['used_usd']:.2f}. "
            f"Resets: {budget['resets_at']}"
        )

# In your LLM call path:
# check_scope_budget("prod-summarization", GATEWAY_URL, GATEWAY_TOKEN)
# response = client.complete(model=..., messages=...)

The "fail closed" pattern matters here: if the budget check is unavailable, you can either allow the request (risking an uncontrolled spend event) or block it. For production patient care workflows that can't tolerate interruption, you may want to fail open instead. Be deliberate about this choice and document it.

Organizational controls

Technical controls fail when employees work around them. The most common workaround in healthcare AI is shadow AI: developers and clinicians using tools that are faster or more capable than the approved path, often without thinking through the compliance implications.

The solution has two parts. First, maintain a clear approved-tools list that includes AI products your developers actually want to use (IDE extensions, coding assistants, chat interfaces) with explicit guidance on BAA status, approved use cases, and the distinction between the consumer product and the enterprise or API version. Second, make the approved path fast enough that it's actually preferred over the workaround.

If a developer has to wait two weeks to get access to an approved AI tool, they'll use an unapproved one. The goal is to remove friction from the compliant path so the workaround isn't necessary, not to enforce compliance by adding friction to the workaround.

The full treatment of shadow AI in healthcare, including a taxonomy of developer tool risk categories and practical governance patterns, is in Shadow AI in healthcare.

Pre-launch checklist

Use this before any AI feature that processes PHI ships to production.

Required before PHI touches the AI stack

[ ] BAA signed with every LLM provider that will receive PHI. Not just the primary provider: if you use multiple models, each provider needs a BAA. See HIPAA-Compliant AI: Business Associate Agreements.

[ ] Audit logging in place and verified. Log prompts, responses, timestamps, model, and user attribution. Test that logging is actually capturing records before going live, not after.

[ ] Logs stored encrypted at rest and access-controlled separately from general application logs.

[ ] Production uses separate API keys from development and staging.

[ ] Model access controls in place: production is restricted to approved, tested models.

[ ] Developers and relevant staff have been told which AI tools are approved and what the approved use cases are.

Security best practice (before Series A or significant PHI volume)

[ ] Scoped keys for each application, environment, and use case. Not just prod vs. dev.

[ ] Security metadata in audit logs: scope, application, environment, duration, token counts.

[ ] Cost controls per scope with hard limits. Budget per scope, not just per organization.

[ ] Log drain to long-term storage configured and tested. HIPAA requires six-year retention; verify that logs are actually reaching the destination.

[ ] Alerting on logging failures: if your audit logging stops working, you need to know before an auditor asks about it.

[ ] De-identification implemented for features that process full clinical notes or free-text PHI.

Defer until you have a dedicated security function

[ ] Automated anomaly detection on LLM usage patterns.

[ ] Penetration testing of LLM-integrated features, specifically for prompt injection in patient-facing surfaces.

[ ] Comprehensive AI tool inventory and enforcement tooling.

[ ] Formal incident response runbook for LLM-specific breach scenarios.

FAQs

Is prompt injection a real concern for my healthcare AI feature?

It depends on what the feature does. Prompt injection is a meaningful threat for patient-facing chat interfaces, agentic workflows where the LLM takes actions based on retrieved content, and RAG pipelines that process externally sourced documents. It's a low-priority threat for internal tools where all input comes from authenticated staff and the model has no action-taking capability.

The OWASP LLM Top 10 covers the full threat taxonomy. The healthcare-specific assessment (when this matters for clinical workflows vs. when it doesn't) is in Prompt injection in healthcare AI.

Do I need to de-identify PHI if I have a BAA?

No. De-identification isn't required when a valid BAA is in place. It's optional additional protection that reduces blast radius if something goes wrong. Whether the added complexity is worth it depends on your use case: features processing clinical notes with full patient context benefit more from de-identification than features working with narrow structured data fields.

See PHI de-identification as a security control for when the tradeoff makes sense and how to implement it for unstructured clinical text.

Can my developers use Claude Code or Cursor with PHI?

The short answer is: only through your covered API agreement and only within the scope of your BAA. The longer answer depends on the tool.

Claude Code uses the Anthropic API. If your organization has a BAA with Anthropic for API usage, and the developer is using Claude Code configured to use your organization's Anthropic API account (not a personal account), that usage is covered. A developer using Claude Code with their personal Anthropic account isn't covered by your organization's BAA. Anthropic's guidance on which products are in scope under a zero data retention agreement confirms Claude Code's API-based usage falls within enterprise BAA coverage.

Cursor uses configurable LLM backends. By default it connects to Anthropic or OpenAI through Cursor's own accounts, not yours. Cursor does offer a Business tier where you can configure your own API keys and enforce a privacy mode.

Before using Cursor with PHI, verify the full data flow rather than just which account handles the LLM request. Which LLM account processes the request is one question; what Cursor's own infrastructure retains along the way is a separate one. Review Cursor's data handling documentation to confirm what is retained at each layer of the request path, and verify that your BAA coverage extends to every system that touches the data.

What is the difference between this guide and the HIPAA compliance guide?

The HIPAA compliance guide covers what HIPAA requires: BAA requirements, audit logging standards, encryption, de-identification under the Privacy Rule, and common implementation mistakes. It's organized around regulatory requirements.

This guide covers what a secure healthcare AI stack needs beyond regulatory compliance: key scoping, security-grade audit logging, organizational governance, cost controls as a security mechanism, and the threat model specific to healthcare operations. It's organized around the security controls that protect the system when something goes wrong, not just when an auditor shows up.

The two guides are complementary. Start with the HIPAA guide if you're getting your compliance baseline in place. This guide is for teams that have done the compliance work and are asking what comes next.

What should I prioritize if I have limited engineering time?

In order of impact-to-effort:

1. Separate production keys from development. This is the highest-leverage, lowest-effort change. If you do one thing from this guide, do this.

2. Verify your audit logging is working. Not "set up logging" (you probably have that). Verify it's actually capturing records. Send a test request, check the logs.

3. Tell your developers which tools are approved. An explicit approved list, including IDE extensions and chat interfaces, prevents shadow AI incidents before they start.

4. Add scope and environment fields to your audit logs. Without attribution, logs aren't useful for investigation or compliance evidence.

5. Set cost limits per scope. This takes 30 minutes to configure and prevents a class of runaway incidents.

Everything else in this guide is worth doing, but these five items have the highest return per hour of engineering time for a small team.

Next steps

Each section of this guide links to a full chapter in the HIPAA AI security guide:

• API key management and scope isolation: full key architecture, rotation patterns, attribution requirements

• Audit logging for healthcare AI: anomaly detection, log drain setup, verifying logging integrity

• PHI de-identification as a security control: NLP-based approaches for unstructured clinical text, token mapping in production

• Shadow AI in healthcare: developer tool governance, approved-tool frameworks, organizational controls

• Prompt injection in healthcare AI: threat model by use case, indirect injection in clinical document processing, mitigations

• Agentic AI in healthcare: security requirements when your LLM takes real-world actions

• Data residency for healthcare AI: GDPR, Australian Privacy Act, and what regional requirements actually mean for LLM infrastructure