assay-server

The Assay server: evaluation and observability for AI systems, meaning document-intelligence pipelines (OCR, classification, splitting, field extraction) and AI agents (reasoning, tool calls, state changes). This package includes the API, the dashboard, and a demo tenant. Python 3.10+.

pip install assay-server
assay demo      # synthetic tenant, 7 weeks of daily runs, staged incidents
assay serve     # http://127.0.0.1:8400  (API docs at /docs)

By default, results go to SQLite in the current directory. For production, add the Postgres driver and point Assay at your database:

pip install "assay-server[postgres]"
export ASSAY_STORE_URL=postgresql+psycopg://user:pass@host:5432/assay

A new server has no login. Creating the first admin key switches login on:

assay keys create --tenant acme --scopes admin --name "acme admin"

To send data to the server from your own code, use the SDK, assay-evals, which has no dependencies.

The setup guide also covers Docker, Vercel, connecting your pipeline's database, and release gates.