anyray-agent

Keeps an agent answering when its Anyray gateway is unreachable. Give your SDK an httpx2 client built on AnyrayTransport(). While the gateway is up, every call goes through it unchanged. When the gateway is down, a call goes straight to the provider instead.

pip install anyray-agent

Python 3.11 or later. One dependency, httpx2, which the current OpenAI and Anthropic SDKs already use.

Use it

Keep the client pointed at your gateway, with your agent key, exactly as before. Add one argument.

import os

import httpx2
from anthropic import Anthropic
from anyray_agent import AnyrayTransport

client = Anthropic(
    base_url=os.environ["ANYRAY_GATEWAY_URL"],
    api_key=os.environ["ANYRAY_CLIENT_KEY"],
    http_client=httpx2.Client(transport=AnyrayTransport()),
)

use_fallback comes from anyray_agent.langchain: ChatAnthropic has no client argument, so it swaps the clients the model builds. Async code uses AsyncAnyrayTransport with httpx2.AsyncClient (http_async_client= on ChatOpenAI, async_transport= on use_fallback). An SDK still on httpx 0.x (openai < 3, anthropic < 2) takes the same classes from anyray_agent.httpx (pip install "anyray-agent[httpx]").

The provider key

A call can only go direct with a provider key on the host: OPENAI_API_KEY and ANTHROPIC_API_KEY are read from the environment, or pass them:

AnyrayTransport(provider_keys={"openai": "…", "anthropic": "…"})

To send Anthropic calls to AWS Bedrock instead of api.anthropic.com, name the region. Credentials come from AWS_BEARER_TOKEN_BEDROCK or the standard AWS variables, or pass api_key or credentials. For an IAM role or a profile, pass boto3's credential chain:

import boto3

AnyrayTransport(bedrock={"region": "us-east-1", "credentials": boto3.Session().get_credentials})

Without a key for a provider, its calls are never sent direct.

When a call goes direct

• Sent direct: the connection to the gateway never opened (refused, DNS, connect timeout) and the gateway's /livez check also fails. After the outage has lasted your org's fallback time (120 seconds by default), calls go direct without trying the gateway first. They return once the gateway has answered for your org's recover time (10 minutes by default).
• Never sent direct: a call that may already have reached the provider (a reset mid-request, any 5xx), so nothing is billed twice. Also a model or route only the gateway serves, and any call while your org has turned fallback off. Nothing goes direct until the agent has read that setting from the gateway.

The gateway key is never sent to a provider.

See what it is doing

transport = AnyrayTransport(on_status=lambda status: print(status.state))
transport.status()  # FallbackStatus(state="armed" | "direct" | "blocked" | "off", reason=..., since=..., calls_direct=...)

The agent also reports its state to the gateway, without any request content, so it shows on the Agents page.

Full reference: docs.anyray.ai/enroll/agent-fallback-reference.

MIT licensed.