generic-ml-cache-daemon

The HTTP daemon for generic-ml-cache. Exposes the cache store and all session/execution functionality as a local REST API with server-sent event (SSE) streaming, plus a caching proxy gateway for the Anthropic Messages API.

Installation

pip install generic-ml-cache-daemon           # runtime only
pip install "generic-ml-cache-daemon[metrics]" # + Prometheus /metrics endpoint

Starting the daemon

python -m generic_ml_cache_daemon             # uses defaults
GMLCACHE_STORE=/path/to/store python -m generic_ml_cache_daemon
GMLCACHE_SESSION=abc GMLCACHE_METRICS=1 python -m generic_ml_cache_daemon

Environment variables:

HTTP API

The daemon listens on http://127.0.0.1:8765 by default. Interactive API docs are available at /docs (Swagger UI) and /redoc.

Observability

Sessions

Executions & Global Stats

Purge body examples:

{"by": "all"}
{"by": "key", "target": "deadbeef"}
{"by": "session", "target": "abc123"}

Run (synchronous or SSE)

POST /run
{
  "client": "anthropic",
  "model": "claude-opus-4-8",
  "prompt": "Summarise the paper.",
  "effort": "medium",
  "session_id": "abc"
}

• Accept: application/json (default) — blocks and returns {execution_key, state, cache_hit, stdout, stderr}
• Accept: text/event-stream — SSE: {"type":"accepted"} immediately, then {"type":"complete", ...} on finish

Jobs (detached / async)

Claude Gateway

POST /gateway/claude/{session_id}/v1/messages

A cache-transparent proxy for the Anthropic Messages API. Requests that hit the cache are returned without a network call to Anthropic. The response is the upstream Anthropic Messages API response returned verbatim — a cache hit is byte-for-byte identical to a live call, with no added fields or headers (the passthrough body must stay exact for SDK compatibility).

⚠️ Security — single-principal, localhost only. The gateway is a single-user tool. Its passthrough cache is keyed on the request body only — deliberately not on the caller's auth token or session (the token refreshes, and keying on it would defeat cross-session cache hits). That is safe for one operator on one machine, but it means two different callers who send the same body share one cache entry regardless of their credentials. Never expose this daemon to a network or to multiple users — a second caller could be served the first caller's provider-authorized response. It binds 127.0.0.1 by default, and a non-loopback --host is refused (a ConfigError) unless you pass --unsafe-allow-network-gateway to loudly accept the risk.

Limitations: single-turn conversations only (one role: user message, no prior assistant turns). Multi-turn support is planned.

Example:

curl http://127.0.0.1:8765/gateway/claude/my-session/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-opus-4-8",
    "messages": [{"role": "user", "content": "Hello, world!"}],
    "max_tokens": 256
  }'

Point any Anthropic SDK client at the gateway by overriding the base URL:

import anthropic

client = anthropic.Anthropic(
    api_key="...",
    base_url="http://127.0.0.1:8765/gateway/claude/my-session",
)

Architecture

The daemon is a thin FastAPI layer over the generic-ml-cache-core hexagonal architecture. It does not own any state — all persistence goes through the existing JournalMetrics (SQLite registry) and ExecutionRepository that the core library manages.

Background jobs run in a ThreadPoolExecutor inside an in-process JobRegistry; job state is not persisted across daemon restarts.

License

Apache-2.0