Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

vigily: keeps watch over your runs, evals and drift

An experiment tracker that keeps watch.
Training runs, evaluations, scheduled evals and drift detection in one self-hosted engine.

Docs · Getting started · Concepts · Python SDK · Drift methods · Changelog · Contributing

The runs table: runs with their status, experiment, parameters and metric values

vigily is in beta: the API may still change before 0.1.0.

Why vigily

Most teams stitch together a tracker for training runs (MLflow, W&B), a tool for LLM and model evaluations (Langfuse) and a monitor for drift (Evidently). vigily is one engine that does the three jobs together, on storage you already have:

  • One engine, no services. vigily-engine (or vigily up) serves the web app, an HTTP API, the scheduler and the drift monitor, and is the only process that writes storage. No database server, no queue, no Node at runtime; the Python SDK is a 60 kB pure-Python package.
  • Evaluations that keep running. Versioned, content-addressed datasets; evaluate() with built-in and custom evaluators; schedules from a config file run them nightly in your time zone.
  • Drift you can trust. Paired tests on matched items with effect sizes and the items that regressed, against a baseline that never absorbs a regression; input drift by PSI and Jensen–Shannon divergence. Every verdict shows its evidence.
  • LLM traces with tokens and cost. vigily.trace(...) records nested spans (also over OTLP/HTTP JSON), prices model calls from your own price table, and stores prompts truncated, in full or not at all, per project.
  • Storage on kv0. Lance tables in a directory or on S3, GCS or Azure; many training jobs can write at once.
  • Fast with many runs and long curves. Thousands of runs filter and sort in memory; charts never ask for more points than they have pixels.

Quickstart

Install with pip

Releases publish two packages to PyPI: vigily, the Python SDK (pure Python, no dependencies), and vigily-engine, the engine that stores runs and serves the web app, as a platform wheel for Linux and macOS. vigily[local] installs both:

pip install 'vigily[local]'   # the SDK and the engine for this machine
vigily-engine                 # serves ./.vigily; open the link it prints (Ctrl-C to stop)
python train.py               # in another shell: logs to that engine

Without a running engine, the SDK starts one in the background for the repository (.vigily, or VIGILY_REPO), which stops after 30 idle minutes. To log to an engine elsewhere, such as a team server, install only the SDK and point it there with a token that has the read and ingest scopes:

pip install vigily
VIGILY_URL=https://vigily.example.com VIGILY_TOKEN=vgy_... python train.py

Until 0.1.0 every release is a beta; pip installs the newest one when no stable release exists.

Install the command line

Each GitHub release also carries prebuilt binaries for Linux (x86_64 and arm64, glibc 2.31 or newer) and macOS (Apple silicon and Intel): vigily, the command line, and vigily-engine. The installer picks the right archive, checks its SHA-256 against the release's checksum file and puts both in ~/.local/bin, without sudo:

curl -LsSf https://raw.githubusercontent.com/sayef/vigily/main/scripts/install.sh | sh

To read the script before running it, download it first:

curl -LsSfO https://raw.githubusercontent.com/sayef/vigily/main/scripts/install.sh
sh install.sh

VIGILY_VERSION=v0.1.0-beta.1 installs a given release and VIGILY_INSTALL_DIR changes the directory. Releases are published as pre-releases, which latest does not serve: set VIGILY_VERSION (or VIGILY_IMAGE for Docker) to a beta tag.

vigily up runs the engine in the foreground and prints a link that signs the browser in; vigily open opens the web app of an engine that is already running, vigily runs lists runs, and vigily tokens create makes tokens for scripts and people on a shared engine (see the command line guide).

Build from source

Build with Rust, Node 24, uv and protoc:

git clone https://github.com/sayef/vigily && cd vigily
(cd crates/vigily-ui && npm ci && npm run build)      # the engine embeds the web app
cargo build --release -p vigily-cli -p vigily-server   # vigily and vigily-engine
uv sync                                               # the Python SDK, editable
export PATH="$PWD/target/release:$PATH"               # the SDK starts vigily-engine from PATH

uv run --no-sync python examples/demo.py              # synthetic training runs
uv run --no-sync python examples/eval_demo.py         # 30 synthetic nights of evaluations
vigily stop && vigily up --config examples/vigily.toml   # open the link it prints

Run with Docker

Each release publishes a multi-platform image (amd64 and arm64) as ghcr.io/sayef/vigily, tagged vX.Y.Z, X.Y and latest. compose.yaml runs it and serves the web app on http://localhost:7600, with the repository in a named volume. The container runs in server mode and needs an admin token, which you also sign in with:

export VIGILY_ADMIN_TOKEN=$(openssl rand -hex 32)
docker compose up -d

compose.dev.yaml builds the image from a checkout instead:

docker compose -f compose.yaml -f compose.dev.yaml up -d --build

Scripts log to it over HTTP with a token that has the read and ingest scopes:

docker compose exec vigily sh -c \
  'vigily --url http://127.0.0.1:7600 --token "$VIGILY_ADMIN_TOKEN" tokens create train --scope read --scope ingest'
VIGILY_URL=http://localhost:7600 VIGILY_TOKEN=vgy_... python train.py

Getting started has the details, and Running a team server covers tokens, TLS and backups.

Log a run:

from vigily import Run

with Run(experiment="cifar10", name="resnet18-lr3e-3") as run:
    run["hparams"] = {"lr": 3e-3, "batch_size": 128}
    for step, loss in train():
        run.track(loss, name="loss", step=step, context={"subset": "train"})

Evaluate a model on a versioned dataset:

from vigily import Dataset, evaluate, evaluators as ev

evaluate(
    Dataset("support-intents"),
    target=classify,
    evaluators=[ev.exact_match(), ev.custom("judge", my_judge)],
    target_ref={"model": "intent-clf-v2"},
    experiment="support-intents-nightly",
)

Screenshots

Loss, accuracy and learning rate for sixteen runs, grouped by optimizer Validation curves as a mean and min-max band per learning rate
Per-item results of an evaluation run with pass and fail verdicts Two evaluation runs compared item by item, regressions first
Drift suites with their verdicts against the baseline Alerts for drift and a failed schedule

The runs list on a phone Charts on a phone in the dark theme

Documentation

The documentation is a mkdocs-material site in docs/; build it with uvx --with mkdocs-material mkdocs serve.

Status

vigily is new (0.1, unreleased). It runs end to end on the demo and on 2 000 synthetic runs with 4.8 million points (start-up 0.2 s, every request in about 0.1 s or less). TODO.md lists what comes next, and docs/kv0-requests.md what it needs from its storage layer.

Drift verdicts and evaluation scores are evidence for people to review, not automated decisions.

Licence

MIT. See LICENSE.

Metadata

Release files for vigily 0.1.0b1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for vigily 0.1.0b1
File Size Uploaded Provenance Actions
vigily-0.1.0b1.tar.gz 73.1 kB With provenance Details for vigily-0.1.0b1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for vigily 0.1.0b1
File Interpreter ABI Platform Actions
vigily-0.1.0b1-py3-none-any.whl Python 3 none any Details for vigily-0.1.0b1-py3-none-any.whl

Total release size: 128.4 kB

Release files / vigily-0.1.0b1.tar.gz

Download URL vigily-0.1.0b1.tar.gz
Size 73.1 kB
Tags Source
SHA-256 checksum
How to use checksums
60b5fd51bc96e89c2c858d766be1bde81e69bbe1e7ed94f8537fc3424c6e6e7e
BLAKE2b-256 checksum
How to use checksums
1220b7bbfac01d295e83f46d6a73865676ea0eef3e30da21b7fcf060ad333d81
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

Release files / vigily-0.1.0b1-py3-none-any.whl

Download URL vigily-0.1.0b1-py3-none-any.whl
Size 55.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
60fe4d63f5bec7cf60a5a2e44c2693105f4bade3a7e44a701b549114af1ee299
BLAKE2b-256 checksum
How to use checksums
a5f8229865a22930bb3bcedffe9d1b64c3ad5fc16b7dc1ff324fe34171332d4e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

Security for vigily 0.1.0b1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for vigily-0.1.0b1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page