title: SimpleAudit Studio emoji: 🔍 colorFrom: blue colorTo: indigo sdk: docker app_port: 7860 pinned: false

SimpleAudit Studio

A self-hostable platform for running reproducible AI model audits using the SimpleAudit engine (Target → Auditor → Judge). Every audit captures frozen, versioned inputs so historical results stay interpretable years later.

🚀 Quick Start

uvx simpleaudit-studio

uvx installs the latest simpleaudit-studio package and runs it in an isolated environment. No Docker, no Postgres, no manual setup. Opens at http://localhost:8000 (login: studio / admin123). A mock model server is pre-seeded so you can start exploring audit results immediately.

🌐 Or skip the setup entirely — try the live demo: sushantgautam-simpleaudit-studio.hf.space

Want the latest unreleased code from this repo instead of the PyPI release? Same one-liner, pointed at git:

uvx --from "git+https://github.com/SushantGautam/SimpleAuditStudio" spin

🤖 Using Real Models

The pre-seeded connections point at a built-in mock server. To run real audits, update your model connections on the Models page to any OpenAI-compatible endpoint. A few common options:

Set the API key in the connection form (leave blank for local servers that don't require auth), then start a run from New Experiment.

🛠️ Local Development

Needs Python 3.11+ and uv. No Docker, no Makefile.

Setup (once)

git clone https://github.com/SushantGautam/SimpleAuditStudio
cd SimpleAuditStudio
cp .env.local.example .env        # SQLite + embedded queue; admin is studio / BOOTSTRAP_PASSWORD
uv sync --extra dev               # create .venv with app + dev tools
uv run manage.py setup_local      # migrate, create admin + workspace, seed scenario packs & models

Daily work

uv run manage.py dev_server --embedded    # web UI + API + worker + embedded Hatchet → http://localhost:8000

Every start applies migrations and makes sure the admin (a superuser) and default workspace exist, so pulling new code needs no extra steps. Keep in mind:

• Only one dev_server --embedded (or uvx simpleaudit-studio) can run at a time: they share the embedded queue database.
• The web server reloads on code changes; the worker does not. Restart dev_server after changing infra/worker.py or infra/engine.py.

Tests and lint

SIMPLEAUDIT_LOCAL_SQLITE=1 uv run manage.py test infra --exclude-tag embedded_hatchet
uv run ruff check .

Other setups

uv run manage.py dev_server --no-worker   # web UI + API only
uv run manage.py dev_server               # use the Postgres + Hatchet configured in .env (see .env.example)

See docs/deployment.md for every environment variable and management command.

🐳 Self-Hosting

For teams or multi-user setups, use Docker Compose:

git clone https://github.com/SushantGautam/SimpleAuditStudio
cd SimpleAuditStudio
cp .env.example .env
# edit POSTGRES_PASSWORD and BOOTSTRAP_PASSWORD at minimum
docker compose up -d

Services: Web UI (:8000), PostgreSQL, Hatchet queue (:8888), Worker. Optional profile: --profile mock (mock model API).

See docs/deployment.md for production hardening, backups, and upgrades.

✨ What You Can Do

• Build versioned scenario sets and register OpenAI-compatible models
• Define versioned judges: start from a SimpleAudit judge (safety, harm, helpfulness, factuality, abstention, checklist, …) and edit its criteria, or write your own criteria as a severity, 1–10 score (with your own dimensions) or yes/no judge; run with any judge model
• Test your deployment's target system prompt, and compare prompts or judges side by side
• Run experiments across models, scenario versions and settings, with a review step before launch
• Watch runs live, including per-repetition results
• Monitor models for drift on a schedule (interval or cron)
• Filter, customise and compare runs on an interactive dashboard

🏗️ Architecture

Browser → Django (gunicorn :8000 in Compose; runserver :7860 in the HF Space / uvx demo)
              ↓
         PostgreSQL 16 (domain + queue DBs)
              ↓
         Hatchet Server (durable job queue)
              ↓
         Worker (CPU pool) → SimpleAudit Engine
              Target → Auditor → Judge

📚 Documentation

• Architecture — system design, service boundaries, deployment topology
• Domain Model — scenarios, scenario sets, audit runs, immutability invariants
• Deployment — Docker Compose, production hardening, upgrades

🤝 Contributing

Architecture decisions are documented in docs/architecture.md.