genruntime 0.4.0

LLM-as-a-Service runtime: reliable, observable, schema-validated inference

GenRuntime

LLM-as-a-Service platform for reliable, observable, schema-validated inference.

---

Why GenRuntime

Problem: Applications that call LLM provider APIs directly (e.g. Anthropic/OpenAI SDK) are fragile — no retries, no output validation, no observability. Failures and debugging are harder in production.

Solution: GenRuntime is a backend layer between your app and LLM providers. It adds retries (with backoff), schema-validated outputs (Pydantic), structured logging (JSON lines), and Prometheus metrics. Use it in code with from genruntime import generate or via HTTP POST /generate. One code path serves both (single source of truth).

---

Features

Feature	Description
Retries	Exponential backoff on transient failures (network, 5xx, rate limit); configurable max retries
Schema validation	generate_structured(prompt, schema=YourModel) — parse JSON and validate with Pydantic; retry once on validation failure
Structured logging	Every request logged as one JSON line (request_id, model, latency_ms, tokens, status); no PII (prompt/response length only)
request_id	Unique ID per request in response and logs for tracing
Health & readiness	GET /health (liveness), GET /ready (config and env check; no provider ping)
Prometheus metrics	GET /metrics — request count, latency histogram, error count (by model/status)
Docker	Dockerfile and docker-compose; run API with env-based config
Multi-provider	Mock (no key), Anthropic Claude, and OpenAI; config-driven via GENRUNTIME_PROVIDER and GENRUNTIME_MODEL

---

Architecture

Request flow: API → runtime → provider adapter → provider. The runtime handles retry, validate, and log. One code path serves both the Python library and the HTTP API (single source of truth).

Applications (Python or HTTP)
         │
         ▼
┌─────────────────────────────────────┐
│  GenRuntime                          │
│  ┌─────────┐  ┌─────────┐  ┌───────┐ │
│  │   API   │→ │ Runtime │→ │ Logger│ │
│  └─────────┘  └────┬────┘  └───────┘ │
│                    │                 │
│              ┌─────▼─────┐           │
│              │ Providers │           │
│              └─────┬─────┘           │
└────────────────────┼─────────────────┘
                     ▼
              LLM providers (Claude, …)

---

Requirements

• Python 3.10+

---

Development setup

Use a virtual environment so project dependencies stay isolated from your system Python and other projects.

1. Create and activate a virtual environment

# Create (run from project root)
python -m venv .venv

# Activate — Windows (PowerShell)
.venv\Scripts\Activate.ps1

# Activate — Windows (CMD)
.venv\Scripts\activate.bat

# Activate — macOS / Linux
source .venv/bin/activate

2. Install the package in editable mode (with dev dependencies)

pip install -e ".[dev]"

This installs genruntime, anthropic, pydantic, fastapi, uvicorn, plus pytest, httpx, ruff for development.

3. Verify

python -c "import genruntime; print(genruntime.__all__)"
pytest tests/unit -v

The .gitignore already excludes .venv and venv/, so the environment is not committed.

---

Quick start

Option 1: Python library

pip install -e .

Set your API key (or use the mock provider without a key):

# Optional: use real Claude (required for GENRUNTIME_PROVIDER=anthropic)
set ANTHROPIC_API_KEY=your_key_here

# Or use OpenAI: set GENRUNTIME_PROVIDER=openai and OPENAI_API_KEY=...
# Or use mock provider (no key needed)
set GENRUNTIME_PROVIDER=mock

Then in Python:

from genruntime import generate

response = generate("Hello, world!")
print(response.text)
print(response.meta.request_id)

Basic usage (with defaults):

from genruntime import generate, configure

configure(model="claude-3-haiku-20240307")

response = generate("Explain neural networks simply")
print(response.text)
print(response.meta.latency_ms)
print(response.meta.request_id)

Structured output:

from pydantic import BaseModel
from genruntime import generate_structured

class Summary(BaseModel):
    topic: str
    summary: str

result = generate_structured("Summarize neural networks", schema=Summary)
print(result.parsed.summary)
print(result.meta.request_id)

Option 2: HTTP API with Docker

Build and run (pass your API key, or use mock):

docker build -t genruntime .
docker run -p 8000:8000 -e ANTHROPIC_API_KEY=your_key_here genruntime

Or with docker-compose (create a .env file with ANTHROPIC_API_KEY=... or set GENRUNTIME_PROVIDER=mock):

docker-compose up --build

Then call the API:

curl -X POST http://localhost:8000/generate ^
  -H "Content-Type: application/json" ^
  -d "{\"prompt\": \"Hello\"}"

On macOS/Linux use \ instead of ^ and single quotes:

curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Hello"}'

Health and readiness:

curl http://localhost:8000/health
curl http://localhost:8000/ready

Optional Prometheus metrics:

curl http://localhost:8000/metrics

---

Configuration

All runtime and API behaviour can be configured via environment variables (12-factor style; no secrets in code).

Variable	Description	Default
ANTHROPIC_API_KEY	API key for Anthropic Claude (required if provider is anthropic)	—
OPENAI_API_KEY	API key for OpenAI (required if provider is openai)	—
GENRUNTIME_PROVIDER	Provider: anthropic, openai, or mock	anthropic if ANTHROPIC_API_KEY set, else openai if OPENAI_API_KEY set, else mock
GENRUNTIME_MODEL	Default model (e.g. claude-3-haiku-20240307, gpt-4o-mini)	claude-3-haiku-20240307
GENRUNTIME_TIMEOUT	Request timeout in seconds	60
GENRUNTIME_MAX_RETRIES	Max retries on transient failure	2

You can also set defaults in code with configure(model=..., timeout=..., max_retries=...) (see Public API below).

---

Public API (library)

The stable surface is 8 items. Everything else is internal.

Item	Description
generate(prompt, **options)	Run inference; returns GenRuntimeResponse (.text, .meta). Options: model, timeout, max_retries, provider.
generate_structured(prompt, schema, **options)	Same as above; parses response as JSON and validates with Pydantic schema. Returns GenRuntimeStructuredResponse (.parsed, .meta). Retries once on validation failure.
configure(**kwargs)	Set default model, timeout, max_retries for function-based usage.
GenRuntime	Class with optional defaults; .generate() and .generate_structured() delegate to the same runtime.
GenRuntimeResponse	Response type: .text (str), .meta (GenRuntimeMeta).
GenRuntimeStructuredResponse	Response type: .parsed (Pydantic model), .meta (GenRuntimeMeta).
GenRuntimeError	Base exception for all runtime failures.
ValidationError	Raised when structured output fails schema validation after retries.

---

Development

Run tests and lint:

pytest tests/unit tests/integration -v
ruff check .