mlproxy-py

mlproxy-py is a minimal ML inference reverse proxy with QoS-aware routing.

Designed for LLM / ML inference workloads where routing decisions should be based on latency, SLA targets, backend health, queue depth, and batching potential.

Features

• Reverse proxy for JSON inference requests
• Backends grouped into model pools
• SLA-aware routing (choose lowest latency backend)
• Optional micro-batching (collect requests for N ms)
• Concurrent health checks with connection pooling
• Prometheus metrics (request count, latency, backend latency)

Quickstart

Install

pip install mlproxy-py

Run proxy

mlproxy run -c examples/config.yml

Send request

curl -X POST http://localhost:7000/infer/modelA \
  -H "Content-Type: application/json" \
  -d '{"text":"hello"}'

Architecture

Client ──POST /infer/{model}──► FastAPI
                                    │
                          ┌─────────▼──────────┐
                          │  ModelRouter       │
                          │  choose_backend()  │
                          │  (score = latency  │
                          │   + active_req*5)  │
                          └─────────┬──────────┘
                                    │ backend URL
                          ┌─────────▼──────────┐
                          │  forward_json()    │
                          │  (httpx conn pool) │
                          └─────────┬──────────┘
                                    ▼
                            Backend ML server

       ┌──────────────────┐    ┌──────────────────┐
       │  BatchQueue      │    │  Healthcheck     │
       │  (optional per   │    │  (concurrent,    │
       │   model pool)    │    │   per-backend)   │
       └──────────────────┘    └──────────────────┘

Config

See examples/config.yml.

Changelog

0.1.1

• Lifespan pattern: Migrated from deprecated @app.on_event("startup") to FastAPI lifespan context manager.
• Graceful shutdown: Batch workers and healthcheck loop are properly cancelled on shutdown.
• Connection pooling: Shared httpx.AsyncClient singletons for proxy and healthcheck (was creating a client per request/check).
• Concurrent health checks: Backends checked in parallel via asyncio.gather (was sequential).
• Logging: Added structured logging throughout; --log-level CLI option.
• Bare except fixes: All except Exception blocks re-raise asyncio.CancelledError.
• Deprecated API fixes: Replaced asyncio.get_event_loop() with asyncio.get_running_loop() in batching module.
• Build system: Migrated from setuptools to hatchling. Added classifiers, keywords, optional dev/test deps, ruff/pytest config.
• Tests: Expanded from 1 test to 15+ tests covering config, router, batching, proxy, healthcheck, and backends.

0.1.0

• Initial release: JSON inference proxy, model pools, SLA-aware routing, micro-batching, health checks, Prometheus metrics.

License

MIT