Agentic AI in healthcare: how to secure LLMs with tool access

By Mat Steinlin, Head of Information Security

Last updated: April 2026

A chat interface that summarizes a clinical note is a read operation. An agent that queries patient records, drafts prior authorization appeals, sends messages to care coordinators, and updates care plans is a read-write system operating in your clinical infrastructure.

That distinction changes the security requirements in kind, not just degree. The blast radius of a misbehaving chat interface is a bad response that a human reads and ignores. The blast radius of a misbehaving agent is actions taken in clinical systems (records updated, messages sent, workflows triggered) before anyone realizes something went wrong.

This chapter covers what the agentic threat model looks like for healthcare AI, the mitigations that matter most, and what HIPAA compliance requires from teams shipping agent-based features. The focus is practical: the frameworks (LangChain, LlamaIndex, CrewAI, OpenAI Assistants API) developers are actually using don't have HIPAA-aware logging or permission scoping built in. This chapter covers what to build on top of them.

What changes when your LLM has tool access

A standard LLM API call is bounded: you send a prompt, you receive a response. The model can produce text that influences downstream decisions, but the model itself doesn't act. A human or application reads the output and decides what to do with it.

When you give a model tool access (function calling, tool use, API integrations), that changes. The model now selects actions and calls tools to execute them. In a healthcare context, those tools may include:

• Querying patient records from an EHR

• Submitting documentation to a payer API

• Sending messages through a patient communication platform

• Writing structured data back to clinical systems

• Triggering downstream workflows

Each tool call is an action in the world. It may be irreversible. It may touch PHI. It may affect other systems and other people. And it happens at model speed: potentially hundreds of actions before any human reviews the output.

The OWASP GenAI agentic threat taxonomy identifies excessive agency as one of the top LLM risks: a model given more capabilities, permissions, or autonomy than the task requires. In healthcare, this isn't an abstract risk category; it's an architectural decision that teams make when defining what tools an agent can call.

Healthcare use cases where agentic AI is being deployed

Understanding the risk profile of your specific agent starts with the use case. These are the patterns most common in healthcare AI today:

Prior authorization

Prior authorization agents query payer APIs, review clinical criteria, identify required documentation, draft appeal letters, and submit packages to payers. This workflow involves reading from clinical records, generating documents that influence care access decisions, and writing to external systems. A compromised or misbehaving prior auth agent doesn't just produce a bad document; it may submit incorrect information to a payer, potentially affecting a patient's access to care.

Care coordination

Care coordination agents query patient records, draft care plans, schedule follow-ups, and send messages to care teams or patients. The communication capabilities are the highest-risk component: an agent that can send messages to patients or external providers has an exfiltration path for PHI and a mechanism for affecting patient behavior.

Clinical documentation

Clinical documentation agents read notes, transcripts, or structured data and write back to EHR fields: problem lists, assessment summaries, structured observations. Write access to clinical records is the most consequential tool capability in healthcare AI. Errors in written clinical data can persist in a patient's record and influence future care decisions by clinicians who don't know the data was AI-generated.

Internal operations

Agents that query internal databases, generate reports, trigger internal workflows, or route tickets don't touch patient-facing clinical systems but may still handle PHI. An internal operations agent that queries a data warehouse containing patient cohort data still has the same HIPAA obligations as a patient-facing feature. Teams sometimes treat internal tooling as lower-stakes; the PHI doesn't know it's internal.

The agentic threat model for healthcare

Prompt injection in agentic context

In a chat interface, a successful indirect prompt injection manipulates the model's text output. In an agent, it can redirect the model's actions. A malicious payload embedded in a clinical note retrieved by a care coordination agent doesn't just produce a manipulated summary; it can redirect the agent to query the wrong record, send a message to an unintended recipient, or call a tool it was not supposed to call for this task.

The attack surface for indirect injection expands with every tool the agent can call. An agent that reads from patient records and writes to an external messaging API can be manipulated into exfiltrating PHI by an adversary who controls content in the retrieval corpus. This isn't theoretical; it's a direct application of the OWASP LLM02 indirect prompt injection pattern to read-write agent architectures.

Excessive permission scope

The most common architectural mistake in healthcare AI agents is giving the agent broader tool access than its task requires. A prior authorization agent that needs to read clinical notes and query payer APIs should not also have write access to the full EHR. A care coordination agent that schedules follow-ups should not have access to billing records.

Excessive permission scope doesn't create vulnerabilities on its own — it amplifies every other threat. A prompt injection that redirects an agent with read-only access to patient notes does limited damage. The same injection against an agent with write access to clinical systems, messaging access to patients, and query access to the full record can do significantly more.

Unaudited agent actions

Under HIPAA's audit control standard (45 CFR 164.312(b)), you must be able to examine activity in systems that contain PHI. This applies to agent tool calls as much as to any other system access.

Most agent frameworks log LLM inputs and outputs. Few log tool calls as first-class auditable events with the attribution, timestamp, and structured metadata that HIPAA investigations require. When an auditor asks "show me all access to Patient X's record between March 10 and March 17," your agent's tool call log needs to be able to answer that question, not just the LLM request log.

Runaway execution

Agents that can loop (calling tools repeatedly based on previous tool outputs) can execute thousands of actions before a rate limit or budget constraint stops them. In general software, a runaway agent process might exhaust API credits. In a healthcare context, it might query thousands of patient records, send hundreds of messages, or write garbage data to clinical fields before anyone notices.

Runaway execution isn't always adversarial. A misconfigured tool definition, an unexpected API response format, or a loop condition in the agent's reasoning can cause it without any external attacker. Rate limiting and action budgets are protective against both.

Data exfiltration via tool outputs

An agent with read access to patient records and any outbound communication capability (messaging, email, webhook, external API write) has a potential exfiltration path for PHI. A successful prompt injection can redirect the agent to read records it shouldn't and send the contents to an unintended destination via a tool it legitimately has access to.

This threat is specific to agents with both read access to PHI and any form of outbound write capability. The mitigation is architectural: minimize the combination of broad read access + outbound communication access in the same agent.

Principle of least privilege for AI agents

The most important mitigation for agentic AI risk is scoping tool access precisely. Each agent should have exactly the tools it needs for its defined task, not a superset of tools that happen to be available.

In practice, this means defining tools at the right granularity. A tool that can "query any patient record" grants more access than a tool that can "query records for the current patient in the session context." A tool that can "send any message via the messaging API" grants more access than a tool that can "send a follow-up message to the patient associated with the current session."

The code example below shows how to define tool schemas that encode permission constraints at the tool definition level, rather than relying on the model to self-limit:

from typing import Any
from dataclasses import dataclass, field

@dataclass
class ScopedTool:
    """
    A tool definition that encodes permission scope explicitly.

    The 'scope' field documents what access the tool requires — used for
    audit logging and access review. The 'readonly' flag controls whether
    the tool is permitted to write to external systems.

    When defining agent tool sets, assemble only the tools the agent
    actually needs for its task — don't pass the full tool registry.
    """
    name: str
    description: str
    parameters: dict
    scope: str            # e.g., "patient_record:read", "messaging:write:current_patient"
    readonly: bool        # If True, this tool cannot modify external state
    requires_approval: bool = False  # If True, human approval required before execution

# Prior authorization agent — reads clinical data, writes to payer API only
PRIOR_AUTH_TOOLS = [
    ScopedTool(
        name="get_patient_clinical_summary",
        description="Retrieve the clinical summary for the current patient, including diagnoses, medications, and relevant notes. Does not return full note text.",
        parameters={
            "type": "object",
            "properties": {
                "patient_id": {"type": "string", "description": "The patient ID for the current session. Must match the session context."},
                "summary_type": {"type": "string", "enum": ["diagnoses", "medications", "recent_notes_summary"], "description": "Which summary component to retrieve."},
            },
            "required": ["patient_id", "summary_type"],
        },
        scope="patient_record:read:summary_only",
        readonly=True,
    ),
    ScopedTool(
        name="submit_prior_auth_request",
        description="Submit a prior authorization request to the payer API. Requires explicit approval before submission.",
        parameters={
            "type": "object",
            "properties": {
                "patient_id": {"type": "string"},
                "payer_id": {"type": "string"},
                "procedure_code": {"type": "string"},
                "clinical_justification": {"type": "string"},
            },
            "required": ["patient_id", "payer_id", "procedure_code", "clinical_justification"],
        },
        scope="payer_api:write:prior_auth",
        readonly=False,
        requires_approval=True,   # Require human review before submitting to payer
    ),
]

def get_tool_schemas_for_provider(tools: list[ScopedTool]) -> list[dict]:
    """
    Convert ScopedTool definitions to the format expected by the LLM provider.
    The provider only sees name, description, and parameters — not scope or
    permission metadata, which stays in your application layer.
    """
    return [
        {
            "name": tool.name,
            "description": tool.description,
            "input_schema": tool.parameters,
        }
        for tool in tools
    ]

Two implementation notes: first, the permission metadata (scope, readonly, requires_approval) lives in your application layer, not in the tool definition sent to the model. The model doesn't need to know about permission scoping; your tool dispatch layer enforces it. Second, assembling per-agent tool sets from a central registry (rather than defining a monolithic tool list) makes it harder to accidentally grant an agent access to tools outside its task scope.

Audit logging for agent actions

Agent tool calls that touch PHI are auditable events under HIPAA. This is distinct from logging the LLM request itself (which the audit logging chapter covers); it requires logging the actions taken in external systems, with enough structured metadata to reconstruct what happened during an investigation.

What a HIPAA-compliant agent action log needs for each tool call:

• Which agent and session initiated the call

• Which tool was called and with what parameters

• Which patient record (or other PHI source) was accessed

• The result status and any errors

• Timestamp and duration

• Whether the action was read-only or modified state

import json
import logging
from datetime import datetime, timezone
from typing import Any, Optional

phi_agent_audit_logger = logging.getLogger("agent.phi_audit")
# This logger MUST write to encrypted, access-controlled storage.
# See the audit logging chapter for log storage requirements.

def execute_tool_with_audit(
    tool: ScopedTool,
    tool_input: dict,
    session_id: str,
    agent_id: str,
    user_id: str,
    tool_executor,           # Callable that actually executes the tool
    requires_approval: bool = False,
    approved: bool = False,
) -> tuple[Any, bool]:
    """
    Execute an agent tool call with HIPAA-compliant audit logging.

    For tools marked requires_approval=True, this function checks that
    explicit approval has been granted before executing. If not approved,
    the call is logged as blocked and returns (None, False).

    Returns (result, success).
    """
    if requires_approval and not approved:
        phi_agent_audit_logger.warning(json.dumps({
            "timestamp":   datetime.now(timezone.utc).isoformat(),
            "event":       "tool_call_blocked_pending_approval",
            "agent_id":    agent_id,
            "session_id":  session_id,
            "user_id":     user_id,
            "tool_name":   tool.name,
            "tool_scope":  tool.scope,
            "tool_input":  tool_input,  # PHI may be present — encrypted storage required
        }))
        return None, False

    start_time = datetime.now(timezone.utc)
    result = None
    status = "success"
    error_detail = None

    try:
        result = tool_executor(tool.name, tool_input)
    except Exception as e:
        status = "error"
        error_detail = type(e).__name__
        raise
    finally:
        phi_agent_audit_logger.info(json.dumps({
            "timestamp":    start_time.isoformat(),
            "event":        "agent_tool_call",
            "agent_id":     agent_id,
            "session_id":   session_id,
            "user_id":      user_id,
            "tool_name":    tool.name,
            "tool_scope":   tool.scope,
            "readonly":     tool.readonly,
            "tool_input":   tool_input,   # PHI may be present — encrypted storage required
            "status":       status,
            "error":        error_detail,
            "duration_ms":  int((datetime.now(timezone.utc) - start_time).total_seconds() * 1000),
        }))

    return result, status == "success"

The tool_input field may contain PHI: patient IDs, clinical parameters, message content. Store this log in the same encrypted, access-controlled storage as your LLM request logs. The same KMS encryption and retention requirements from chapter 3 apply.

Controlled execution patterns

Human-in-the-loop for write operations

For agent actions that write to clinical systems or communicate with patients, require explicit human approval before execution. This is operationally costly (it removes the automation benefit for those actions), but it's the right trade-off when the consequences of an error are patient-facing.

The requires_approval flag in the tool definition above implements this at the tool dispatch level. Your application layer presents the proposed action to a human reviewer, records the approval decision, and only then calls execute_tool_with_audit with approved=True.

What approvals to require is a product and risk decision. A reasonable starting framework:

• Clinical record writes: require approval

• Patient-facing communications: require approval

• Payer API submissions: require approval

• Read operations and internal queries: no approval required unless they involve full-record access

Dry run mode

Before deploying an agent to production, test it against realistic inputs in a mode where tool calls are simulated rather than executed. Log what the agent would have done (the sequence of tool calls, the parameters, the order of operations) and review the results for unexpected behavior before real execution is permitted.

Dry run testing is also useful for adversarial testing: inject prompt payloads into the simulated tool outputs and observe whether the agent's behavior changes in ways that indicate susceptibility.

Action budgets and rate limiting

Every agent execution should have a maximum number of tool calls it can make within a single run. This prevents runaway loops from executing indefinitely while staying within your monitoring window.

from dataclasses import dataclass, field

@dataclass
class ActionBudget:
    """
    Per-execution action budget for an agent.

    Call check_and_consume() before each tool execution. If the budget
    is exhausted, raise an exception that terminates the agent run
    and triggers a logged alert.
    """
    max_tool_calls: int
    max_write_calls: int          # Subset limit for write operations
    max_calls_per_tool: dict[str, int] = field(default_factory=dict)
    _total_calls: int = field(default=0, init=False)
    _write_calls: int = field(default=0, init=False)
    _calls_per_tool: dict[str, int] = field(default_factory=dict, init=False)

    def check_and_consume(self, tool: ScopedTool) -> None:
        """
        Check budget before executing a tool call and consume one unit.
        Raises BudgetExhaustedError if any limit is exceeded.

        Call this inside execute_tool_with_audit before the tool executes.
        """
        if self._total_calls >= self.max_tool_calls:
            raise BudgetExhaustedError(
                f"Agent execution exceeded max_tool_calls limit of {self.max_tool_calls}. "
                f"Terminating run to prevent runaway execution."
            )

        if not tool.readonly and self._write_calls >= self.max_write_calls:
            raise BudgetExhaustedError(
                f"Agent execution exceeded max_write_calls limit of {self.max_write_calls}."
            )

        per_tool_limit = self.max_calls_per_tool.get(tool.name)
        tool_count = self._calls_per_tool.get(tool.name, 0)
        if per_tool_limit is not None and tool_count >= per_tool_limit:
            raise BudgetExhaustedError(
                f"Tool '{tool.name}' call limit of {per_tool_limit} exceeded in this run."
            )

        self._total_calls += 1
        if not tool.readonly:
            self._write_calls += 1
        self._calls_per_tool[tool.name] = tool_count + 1

class BudgetExhaustedError(Exception):
    """Raised when an agent run exceeds its action budget."""
    pass

# Example: prior authorization agent budget
PRIOR_AUTH_BUDGET = ActionBudget(
    max_tool_calls=20,
    max_write_calls=3,      # At most 3 payer API submissions per run
    max_calls_per_tool={
        "submit_prior_auth_request": 1,    # Submit once, not in a loop
        "get_patient_clinical_summary": 5, # Allow a few retrieval steps
    },
)

Budget limits should be set based on what the agent should normally need to complete its task, with a safety margin. An agent that needs 3–5 tool calls to complete its task should have a budget of 10–15, not 1,000. A BudgetExhaustedError should be logged as an anomaly, not silently swallowed; it indicates either a runaway condition or an adversarial input worth investigating.

Testing agentic AI before deployment

Agent behavior is harder to test than chat interface behavior because the output is a sequence of actions, not a single response. Standard LLM evals don't capture whether the agent called the right tools in the right order with the right parameters.

Before deploying an agent to production clinical systems:

Test expected task completion. Run the agent against a representative set of realistic inputs in a staging environment with production-equivalent (but de-identified) data. Verify that the tool call sequence matches what you expect for each input.

Test boundary conditions. What happens when the agent encounters unexpected API responses, missing data, or edge case inputs? A prior auth agent that receives a malformed payer API response should fail gracefully, not loop or call additional tools trying to recover.

Test adversarial inputs. Inject prompt injection payloads into tool outputs (the content that returns from your EHR query, the payer API response, the patient message content) and verify that the agent's behavior doesn't change in ways that indicate susceptibility. This is the agentic extension of the injection testing described here.

Verify permission constraints. Confirm that the agent can't call tools outside its defined tool set, even if a prompt payload attempts to reference them. Your tool dispatch layer should enforce this; the test verifies it does.

Review the audit log. For each test run, review the action audit log and confirm that every tool call is captured with complete, accurate metadata. The audit log is only useful for HIPAA investigations if it reliably captures what the agent actually did.

What this means for HIPAA compliance

Agentic workflows don't create new HIPAA requirements; they expand the audit surface.

Every tool call that accesses, modifies, or transmits PHI is subject to the same requirements as any other PHI-processing operation: it must be logged, the log must be retained per your retention policy, access to the log must be controlled, and you must be able to produce a record of the activity during an audit or breach investigation.

What this means operationally:

BAA coverage applies to each tool endpoint. If your agent calls a third-party API that handles PHI (a payer API, a patient communication platform, an external data service), that vendor must have a BAA with you. The agent calling the API doesn't change the BAA requirement; it just means there are more potential BAA gaps to close. See the BAA chapter for what BAA coverage actually requires.

Agent logs are PHI if they contain patient data. Tool call logs that include patient IDs, clinical parameters, or message content are PHI. They must be encrypted at rest, access-controlled, and retained according to your PHI retention policy, the same as your LLM request logs.

Automated actions are still your organization's actions. An agent submitting incorrect information to a payer or sending a message to the wrong patient is your organization taking that action, not the vendor's. HIPAA obligations and breach notification requirements apply regardless of whether a human or an AI was the proximate cause.

Teams using agent frameworks like LangChain, LlamaIndex, or OpenAI's Assistants API should treat the framework as infrastructure that provides agent orchestration, not as infrastructure that provides HIPAA compliance. The logging, permission scoping, action budgets, and audit controls described in this chapter are what you build on top.

FAQs

Do I need human approval for every agent action?

No. You do need human approval for any action where the consequences of an error are patient-facing and hard to reverse. A read-only query that informs a draft document that a human reviews doesn't need individual approval. A write to a clinical record, a message sent to a patient, or a submission to an external payer does. The principle is: the more irreversible the action and the more directly it affects patient care, the stronger the case for requiring explicit approval.

Can I use multi-agent frameworks (CrewAI, AutoGen) in healthcare applications?

Multi-agent systems introduce additional complexity: one agent's output becomes another agent's input, and a compromise or misconfiguration at one layer propagates. The same principles apply: each agent should have scoped tool access, and every PHI-touching tool call should be logged. The practical challenge is that multi-agent frameworks have even less HIPAA-aware infrastructure than single-agent frameworks. If you're using a multi-agent pattern for healthcare workloads, the audit logging and permission scoping need to happen at each agent in the chain, not just at the entry point.

How does this interact with the AI Gateway?

Aptible AI Gateway covers the LLM request layer of an agent run: BAA coverage, scoped key attribution, and request/response logging for every call the agent makes to the model. In a multi-step reasoning loop, each step's LLM call is logged and attributed through the gateway automatically.

What the gateway doesn't cover is the external tool calls your agent makes: EHR queries, payer API calls, patient messaging. Those connections run directly from your application to those systems, and the audit logging for them is your responsibility, as described earlier in this chapter.

The practical split: gateway handles the model side; your application handles the tool call side. Both need to be in place for a complete audit trail.

Next steps

If your product serves EU, Australian, or Canadian markets, the agentic controls in this chapter need to operate within regional data residency constraints — which affects where your gateway, model endpoints, and audit logs can live.

• Data residency for healthcare AI: what HIPAA, GDPR, the Australian Privacy Act, and Canadian law each require at the infrastructure level

• Prompt injection in healthcare AI: the injection threat model that feeds directly into agentic security, if you haven't read it yet