US-only inference for all models

To support customers with strict data residency requirements, LLM Gateway now guarantees US-only inference for all models. We also have begun documenting the data retention mode for every model, to clarify which models guarantee zero data retention (ZDR), and which require private data retention for up to 30 days, for security review. To confirm the data retention mode for models you access through LLM Gateway, see Supported Models.

Spend alert emails

Now, whenever spend reaches 90% of either the global spend limit or any key-level spend limit, Account Owners and Key Owners will be immediately notified. A second notification will be sent once the spend limit is reached. To learn more about global and per-key spend limits, see Cost Visibility & Control.

Prompt caching improvements

In order to support high availability even during LLM provider outages, LLM Gateway automatically fails over to alternate providers of the same model without any action required from you. Previously, this meant that in some cases, our provider routing might result in prompt cache misses even in cases when a cache hit should have been possible. We've updated our provider routing logic to ensure that LLM requests are routed to a provider with a warm cache except in cases where an outage makes this impossible.

To take full advantage of prompt caching, your client should send a session ID with each of its requests. Most agent harnesses (Claude Code, Codex, OpenCode and Pi) send session IDs automatically. If you're using the Anthropic or OpenAI SDK directly, include a session-id header in your requests. For more information including example code, see Prompt Caching.