ttllm-gateway 0.0.10

LLM Gateway with Anthropic-compatible API

TTLLM Gateway

LLM gateway exposing an Anthropic-compatible API (POST /v1/messages), routing requests through LangChain to any supported provider (Bedrock, OpenAI, etc.). Tracks tokens, costs, and maintains audit trails. Supports user management with per-user model access control.

Quick Start

Prerequisites

• Python 3.12+
• PostgreSQL 16
• Docker (optional)

Run with Docker Compose

docker-compose up

This starts PostgreSQL and the API on port 8000. Migrations run automatically on container start.

A default admin account is created by the migrations:

• Email: admin@localhost
• Password: value of TTLLM_ADMIN_PASSWORD (defaults to admin)

Set TTLLM_ADMIN_PASSWORD before running migrations to use a custom password. Log in via ttllm login and change the password or create a new admin user immediately.

Run from Docker Image

# From GitHub Container Registry
docker run -p 8000:8000 \
  -e TTLLM_DATABASE__URL="postgresql+asyncpg://user:pass@host:5432/ttllm" \
  ghcr.io/ponquersohn/ttllm-gateway:latest

Passing configuration

Option A - Mount a config file:

docker run -p 8000:8000 \
  -v /path/to/config.yaml:/app/config.yaml \
  -e TTLLM_CONFIG_FILE=/app/config.yaml \
  -e TTLLM_CONFIG_ENV=prod \
  ghcr.io/ponquersohn/ttllm-gateway:latest

Option B - Environment variables only:

docker run -p 8000:8000 \
  -e TTLLM_DATABASE__URL="postgresql+asyncpg://user:pass@host:5432/ttllm" \
  -e TTLLM_AUTH__JWT__SECRET_KEY="your-secret" \
  -e TTLLM_ENGINE__LISTEN_PORT=8000 \
  -e TTLLM_PROVIDER__DEFAULT_REGION="us-east-1" \
  ghcr.io/ponquersohn/ttllm-gateway:latest

The container listens on port 8000 by default (configurable via engine.listen_port). Map it to any host port with -p <host>:<container>.

Debugging failed containers

By default the container exits on error. Set TTLLM_EXIT_ON_ERROR=false to keep the container alive after a failure, so you can exec into it for debugging:

docker run -p 8000:8000 \
  -e TTLLM_EXIT_ON_ERROR=false \
  -e TTLLM_DATABASE__URL="postgresql+asyncpg://user:pass@host:5432/ttllm" \
  ghcr.io/ponquersohn/ttllm-gateway:latest

Install from PyPI

pip install ttllm-gateway

Run Locally

pip install -e .
alembic upgrade head
uvicorn ttllm.handlers.ecs_entrypoint:app --reload

Configuration

Settings are resolved in order: YAML config file -> environment variables -> defaults.

Environment Variable	Description	Default
TTLLM_CONFIG_FILE	Path to YAML config file	(none)
TTLLM_CONFIG_ENV	Environment section to load	dev
TTLLM_DATABASE__URL	PostgreSQL connection string	postgresql+asyncpg://ttllm:dev@localhost:5432/ttllm
TTLLM_ENGINE__LISTEN_PORT	Server listen port	8000
TTLLM_ENGINE__BASE_URL	External-facing URL (for OAuth callbacks)	http://localhost:4000
TTLLM_ENGINE__CORS_ORIGINS	Allowed CORS origins	["*"]
TTLLM_AUTH__JWT__SECRET_KEY	JWT signing secret	CHANGE-ME-IN-PRODUCTION
TTLLM_PROVIDER__DEFAULT_REGION	AWS region for Bedrock	us-east-1
TTLLM_SECRETS__ENCRYPTION_KEY	Fernet key for encrypting secrets	(none)

Nested env vars use __ as delimiter. YAML values support env://VAR,default and secret://arn resolution patterns. Local overrides via local.config.yaml (git-ignored).

Config file example

dev:
  database:
    url: "postgresql+asyncpg://ttllm:dev@localhost:5432/ttllm"
    pool_size: 5
  engine:
    base_url: "http://localhost:8000"
    listen_port: 8000
    cors_origins: ["*"]
    log_request_bodies: false
  auth:
    jwt:
      secret_key: "dev-secret"
      algorithm: "HS256"
      access_token_ttl_minutes: 15
    identity_providers:
      entra:
        name: "Entra ID"
        type: "oidc"
        client_id: "..."
  provider:
    default_region: "us-east-1"
  secrets:
    encryption_key: "env://TTLLM_SECRETS_ENCRYPTION_KEY"

Secrets Management

Provider credentials (AWS keys, API keys, etc.) can be stored encrypted in the database and referenced from model configs using secret://name. This avoids storing plaintext credentials in config_json.

Setup

1. Generate an encryption key and add it to your config:

python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

Add to config.yaml:

dev:
  secrets:
    encryption_key: "your-generated-key"

Or via environment variable: TTLLM_SECRETS__ENCRYPTION_KEY.

2. Create secrets:

ttllm secrets create --name aws-bedrock-key        # prompts for value (hidden)
ttllm secrets create --name aws-bedrock-secret      # prompts for value (hidden)

3. Reference secrets in model config:

ttllm models create \
  --name claude-sonnet \
  --provider bedrock \
  --provider-model-id anthropic.claude-3-sonnet-20240229-v1:0 \
  --config '{"aws_access_key_id":"secret://aws-bedrock-key","aws_secret_access_key":"secret://aws-bedrock-secret","region":"us-west-2"}'

At runtime, secret:// references are resolved transparently before the provider client is created. Secret values are never exposed through the API or CLI.

CLI

Admin operations via the ttllm CLI:

ttllm status                         # Show server version and status
ttllm users list|show|create|update|delete
ttllm models list|show|create|update|delete|assign|unassign
ttllm groups list|show|create|update|delete
ttllm tokens list|show|create|delete
ttllm secrets list|show|create|update|delete
ttllm usage summary|costs [--user] [--model] [--since] [--until]
ttllm audit-logs [--user] [--model] [--limit]

Releasing

Releases are created from the main branch. The Makefile bumps the version in src/ttllm/__init__.py and shows the commands to complete the release:

make release         # Patch bump (v0.0.1 -> v0.0.2)
make release-minor   # Minor bump (v0.1.0 -> v0.2.0)
make release-major   # Major bump (v1.0.0 -> v2.0.0)

After running make release*, follow the printed instructions to commit, tag, push, and create the GitHub release. Publishing a GitHub release triggers the CI workflow to:

1. Validate that the git tag matches the __version__ in code
2. Publish the Python package to PyPI
3. Build and push the Docker image to ghcr.io/ponquersohn/ttllm-gateway

Development

pip install -e ".[dev]"
pytest