LLM Observer Proxy

LLM Observer Proxy runs a temporary LiteLLM data plane for each evaluation or agent run and stores its observable LLM traffic under a stable run_id. It records requests, responses, streaming output, tool calls, token usage, cost estimates, errors, and provider-visible reasoning content.

It is independently installable and can be embedded into any evaluation platform, agent runner, or orchestration service through its HTTP API.

How It Works

There are two HTTP layers:

1. The control plane listens on port 8790 by default. An orchestrator uses it to create, inspect, and stop run-scoped proxies.
2. Each data plane listens on a separate, automatically allocated port. An agent uses the returned URL as an OpenAI-compatible or Anthropic-compatible LLM endpoint.

One control-plane process can manage multiple concurrent data planes. Stopping a data plane preserves its history on disk, and history remains readable after the control plane restarts.

The observer can only record information exposed by the upstream API. It cannot obtain hidden chain-of-thought that a provider does not return.

Requirements

• Python 3.11, 3.12, or 3.13
• A provider API key or a custom LiteLLM configuration
• Local permission to start subprocesses and bind data-plane ports

Install

Install the command in the current Python environment:

python -m pip install llm-observer-proxy

Or install it as an isolated command with uv:

uv tool install llm-observer-proxy

Run it directly without a permanent installation:

OPENAI_API_KEY=your-openai-api-key uvx llm-observer-proxy

Because the distribution name and console command are both llm-observer-proxy, uvx resolves the package and starts the server in one command. It caches the isolated environment for subsequent runs.

For a source checkout, install the locked development environment instead:

uv sync --extra dev

LiteLLM is pinned because the observer callback adapts specific response and streaming internals. The package declares the subset of LiteLLM Proxy runtime dependencies needed by the tested LLM data plane rather than installing the full litellm[proxy] extra.

Quick Start

The bundled catalog includes gpt-5.5 and reads the OpenAI credential from the environment:

export OPENAI_API_KEY=your-openai-api-key
llm-observer-proxy

In another terminal, check the control plane:

curl http://127.0.0.1:8790/healthz

Create a data plane for a run:

curl -X POST http://127.0.0.1:8790/api/runs/run-001/proxy \
  -H 'Content-Type: application/json' \
  -d '{}'

The data plane serves every model in the selected catalog or custom LiteLLM configuration. Each LLM request selects a route through its own model field. The create response includes a dynamically allocated bare base_url, for example http://127.0.0.1:45678, plus openai_base_url (http://127.0.0.1:45678/v1) and anthropic_base_url (http://127.0.0.1:45678). Use the explicit URL for the client SDK, or call an endpoint directly:

curl http://127.0.0.1:45678/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-5.5",
    "messages": [{"role": "user", "content": "Reply with one short sentence."}]
  }'

Read the normalized history:

curl http://127.0.0.1:8790/api/runs/run-001/history

Stop the data plane without deleting its history:

curl -X DELETE http://127.0.0.1:8790/api/runs/run-001/proxy

Configuration

The command accepts these options:

--host HOST                 Control-plane bind host (default: 127.0.0.1)
--port PORT                 Control-plane port (default: 8790)
--env-file PATH             KEY=VALUE file loaded before startup (default: .env)
--artifact-root PATH        Artifact storage root
--model-catalog PATH        Model catalog JSON
--log-level LEVEL           Python log level

Paths are resolved in this order:

1. --artifact-root and --model-catalog
2. LLM_OBSERVER_PROXY_ARTIFACT_ROOT and LLM_OBSERVER_PROXY_MODEL_CATALOG_PATH
3. ./artifacts and ./config/models.json in the working directory
4. The package's secret-free bundled model catalog

Relative paths are resolved against the process working directory. LITELLM_BIN can select a specific LiteLLM executable.

Model Catalog

A model catalog has this shape:

{
  "llm_observer_proxy": {
    "base_url": "https://api.openai.com",
    "api_key": "os.environ/OPENAI_API_KEY",
    "litellm_settings": {
      "drop_params": true
    }
  },
  "models": [
    {
      "id": "gpt-5.5",
      "upstream_api": "openai"
    }
  ]
}

id is the public model alias exposed by the data plane. upstream_api selects LiteLLM's openai or anthropic provider adapter. Optional input and output token prices are used for local cost estimates. Other catalog properties, including name and provider, are accepted but ignored. Environment references use LiteLLM's os.environ/NAME syntax, which prevents a literal credential from being written to the generated configuration. See config/models.example.json for a complete example.

For advanced provider routing, pass a server-local LiteLLM YAML path as litellm_config_path in the create request. The observer callback is added to that configuration automatically.

Control API

The create payload may be empty. Optional fields are port, bind_host, litellm_config_path, litellm_env, and strip_callbacks. Unknown fields are ignored. The complete request and response contract is documented in docs/control-plane-api.md.

Artifacts

Each run writes:

artifacts/runs/<run_id>/platform/llm/
  config.yaml
  conversation_snapshot.json
  events.jsonl
  litellm.log

The generated YAML is retained as a run artifact. Keep provider credentials in environment variables rather than writing literal values into a model catalog or custom YAML.

Supported Scope

The supported data-plane surface is LiteLLM's OpenAI-compatible LLM API and Anthropic /v1/messages, plus the control and history APIs documented above. LiteLLM MCP routes, its administration UI, database persistence, and enterprise gateway features are not part of this project's compatibility contract. In particular, the mcp package is intentionally not installed because MCP routes are not used by the observer and are lazy-loaded by the pinned LiteLLM version.

Troubleshooting

• The control plane starts, but proxy creation returns 502: inspect the returned log_path or the run's litellm.log. Missing provider variables and invalid custom YAML are the most common causes.
• LiteLLM cannot be found: install the package and command in the same environment, or set LITELLM_BIN to an executable path.
• Proxy creation returns 409: the run_id already has a running data plane, a requested port is unavailable, or the selected configuration is invalid.
• History is empty: confirm the agent is using the returned data-plane URL, not the provider URL directly.

Security

The control API is an administrative interface: it can start subprocesses, select listening ports, pass child environment values, and read server-local LiteLLM configurations. Version 0.1.x has no built-in authentication. Bind it to loopback or a trusted private network and read .github/SECURITY.md before remote deployment.

History can contain prompts, model output, tool output, and credentials. Apply appropriate filesystem permissions, access control, and retention policies to the artifact root.

Development

uv sync --extra dev
uv run pytest
LLM_OBSERVER_PROXY_RUN_INTEGRATION=1 uv run pytest tests/test_integration_proxy.py
uv build
uvx twine check dist/*

License

Apache-2.0. See LICENSE and NOTICE.

Authors

• Shuqiao Zhang stevenjoezhang@gmail.com
• Yue Xiong xxyyyue@gmail.com