Matyan Frontier

Ingestion gateway between training clients and the rest of the Matyan stack. Clients connect via WebSocket for metrics, params, and logs, and via REST for presigned URLs (S3, GCS, or Azure Private SAS); the frontier publishes to Kafka. The UI and backend do not talk to the frontier. Part of the Matyan experiment-tracking stack (fork of Aim).

Layout

• src/matyan_frontier/ — Python package: FastAPI app, WebSocket handler, REST presign endpoint, Kafka producer, config, health, metrics.
• Entrypoints: app.py (lifespan: start Kafka producer, create blob storage clients, ensure bucket; shutdown: flush producer, close clients).
• Routes: GET /api/v1/ws/runs/{run_id} (WebSocket), POST /api/v1/rest/artifacts/presign, GET /health/ready/, GET /health/live/, GET /metrics/ (Prometheus).

Prerequisites

• Python 3.12+. The package uses uv in the repo: uv run matyan-frontier or install then matyan-frontier CLI.
• Runtime dependencies: Kafka (bootstrap reachable) and an blob store (e.g. RustFS in dev, AWS S3/GCS/Azure Blob Store in prod). The smoke test and local dev assume Kafka and S3 are up (e.g. via docker-compose).

Run (production-like)

From the frontier package directory: uv run matyan-frontier start (or matyan-frontier start if installed).

Options: --host, --port (defaults: 0.0.0.0, 53801). The CLI uses these option defaults; config also defines host/port for other entry points.

Configuration (environment variables)

Source of truth: config.py.

Development and smoke test

• Development: Run Kafka and S3 (e.g. docker compose up kafka kafka-init rustfs or equivalent), then uv run matyan-frontier start. Clients (e.g. matyan-client) point at the frontier URL for WebSocket and presign.
• Smoke test: From extra/matyan-frontier, run uv run python scripts/smoke_test.py. Prerequisites: Kafka, S3, and frontier running. The script covers WebSocket (create_run, log_metric, log_hparams, finish_run, etc.) and REST presign, and optionally verifies Kafka consumption.

See scripts/smoke_test.py for the exact command and what it checks.

Deployment

• Docker: Use Dockerfile.dev or Dockerfile.prod (context from repo root as needed; align with how the repo builds frontier images).

The UI talks only to the backend, not to the frontier; the frontier is for training clients.

Related

• Backend: matyan-backend serves the REST API and consumes from Kafka (ingestion + control workers); it does not receive client traffic from the frontier directly.
• Client: matyan-client sends tracking data to the frontier (WebSocket + presign REST).
• Monorepo: This package lives under extra/matyan-frontier in the matyan-core repo.