This release is a pre-release and may not be stable for production use.
An experiment tracker that keeps watch.
Training runs, evaluations, scheduled evals and drift detection in one self-hosted engine.
Docs · Getting started · Concepts · Python SDK · Drift methods · Changelog · Contributing
vigily is in beta: the API may still change before 0.1.0.
Why vigily
Most teams stitch together a tracker for training runs (MLflow, W&B), a tool for LLM and model evaluations (Langfuse) and a monitor for drift (Evidently). vigily is one engine that does the three jobs together, on storage you already have:
- One engine, no services.
vigily-engine(orvigily up) serves the web app, an HTTP API, the scheduler and the drift monitor, and is the only process that writes storage. No database server, no queue, no Node at runtime; the Python SDK is a 60 kB pure-Python package. - Evaluations that keep running. Versioned, content-addressed datasets;
evaluate()with built-in and custom evaluators; schedules from a config file run them nightly in your time zone. - Drift you can trust. Paired tests on matched items with effect sizes and the items that regressed, against a baseline that never absorbs a regression; input drift by PSI and Jensen–Shannon divergence. Every verdict shows its evidence.
- LLM traces with tokens and cost.
vigily.trace(...)records nested spans (also over OTLP/HTTP JSON), prices model calls from your own price table, and stores prompts truncated, in full or not at all, per project. - Storage on kv0. Lance tables in a directory or on S3, GCS or Azure; many training jobs can write at once.
- Fast with many runs and long curves. Thousands of runs filter and sort in memory; charts never ask for more points than they have pixels.
Quickstart
Install with pip
Releases publish two packages to PyPI: vigily, the Python SDK (pure Python, no dependencies),
and vigily-engine, the engine that stores runs and serves the web app, as a platform wheel for
Linux and macOS. vigily[local] installs both:
pip install 'vigily[local]' # the SDK and the engine for this machine
vigily-engine # serves ./.vigily; open the link it prints (Ctrl-C to stop)
python train.py # in another shell: logs to that engine
Without a running engine, the SDK starts one in the background for the repository (.vigily, or
VIGILY_REPO), which stops after 30 idle minutes. To log to an engine elsewhere, such as a
team server, install only the SDK and point it there with a token
that has the read and ingest scopes:
pip install vigily
VIGILY_URL=https://vigily.example.com VIGILY_TOKEN=vgy_... python train.py
Until 0.1.0 every release is a beta; pip installs the newest one when no stable release exists.
Install the command line
Each GitHub release also carries prebuilt binaries for Linux (x86_64 and arm64, glibc 2.31 or
newer) and macOS (Apple silicon and Intel): vigily, the command line, and vigily-engine. The
installer picks the right archive, checks its SHA-256 against the release's checksum file and puts
both in ~/.local/bin, without sudo:
curl -LsSf https://raw.githubusercontent.com/sayef/vigily/main/scripts/install.sh | sh
To read the script before running it, download it first:
curl -LsSfO https://raw.githubusercontent.com/sayef/vigily/main/scripts/install.sh
sh install.sh
VIGILY_VERSION=v0.1.0-beta.1 installs a given release and VIGILY_INSTALL_DIR changes the
directory. Releases are published as pre-releases, which latest does not serve: set
VIGILY_VERSION (or VIGILY_IMAGE for Docker) to a beta tag.
vigily up runs the engine in the foreground and prints a link that signs the browser in;
vigily open opens the web app of an engine that is already running, vigily runs lists runs,
and vigily tokens create makes tokens for scripts and people on a shared engine (see the
command line guide).
Build from source
Build with Rust, Node 24, uv and protoc:
git clone https://github.com/sayef/vigily && cd vigily
(cd crates/vigily-ui && npm ci && npm run build) # the engine embeds the web app
cargo build --release -p vigily-cli -p vigily-server # vigily and vigily-engine
uv sync # the Python SDK, editable
export PATH="$PWD/target/release:$PATH" # the SDK starts vigily-engine from PATH
uv run --no-sync python examples/demo.py # synthetic training runs
uv run --no-sync python examples/eval_demo.py # 30 synthetic nights of evaluations
vigily stop && vigily up --config examples/vigily.toml # open the link it prints
Run with Docker
Each release publishes a multi-platform image (amd64 and arm64) as ghcr.io/sayef/vigily, tagged
vX.Y.Z, X.Y and latest. compose.yaml runs it and serves the web app on
http://localhost:7600, with the repository in a named volume. The container runs in server mode
and needs an admin token, which you also sign in with:
export VIGILY_ADMIN_TOKEN=$(openssl rand -hex 32)
docker compose up -d
compose.dev.yaml builds the image from a checkout instead:
docker compose -f compose.yaml -f compose.dev.yaml up -d --build
Scripts log to it over HTTP with a token that has the read and ingest scopes:
docker compose exec vigily sh -c \
'vigily --url http://127.0.0.1:7600 --token "$VIGILY_ADMIN_TOKEN" tokens create train --scope read --scope ingest'
VIGILY_URL=http://localhost:7600 VIGILY_TOKEN=vgy_... python train.py
Getting started has the details, and Running a team server covers tokens, TLS and backups.
Log a run:
from vigily import Run
with Run(experiment="cifar10", name="resnet18-lr3e-3") as run:
run["hparams"] = {"lr": 3e-3, "batch_size": 128}
for step, loss in train():
run.track(loss, name="loss", step=step, context={"subset": "train"})
Evaluate a model on a versioned dataset:
from vigily import Dataset, evaluate, evaluators as ev
evaluate(
Dataset("support-intents"),
target=classify,
evaluators=[ev.exact_match(), ev.custom("judge", my_judge)],
target_ref={"model": "intent-clf-v2"},
experiment="support-intents-nightly",
)
Screenshots
Documentation
The documentation is a mkdocs-material site in docs/; build it with
uvx --with mkdocs-material mkdocs serve.
- Getting started and concepts
- Guides: Python SDK, command line, configuration, scheduled evaluations and security, drift methods, LLM tracing, self-hosting and storage, the audit trail
- Design: brand, architecture, decisions
- FAQ
Status
vigily is new (0.1, unreleased). It runs end to end on the demo and on 2 000 synthetic runs with 4.8 million points (start-up 0.2 s, every request in about 0.1 s or less). TODO.md lists what comes next, and docs/kv0-requests.md what it needs from its storage layer.
Drift verdicts and evaluation scores are evidence for people to review, not automated decisions.
Licence
MIT. See LICENSE.
Metadata
Release files for vigily 0.1.0b1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| vigily-0.1.0b1.tar.gz | 73.1 kB | With provenance | Details for vigily-0.1.0b1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| vigily-0.1.0b1-py3-none-any.whl | Python 3 | none | any | Details for vigily-0.1.0b1-py3-none-any.whl |
Total release size: 128.4 kB
Release files / vigily-0.1.0b1.tar.gz
| Download URL | vigily-0.1.0b1.tar.gz |
|---|---|
| Size | 73.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
60b5fd51bc96e89c2c858d766be1bde81e69bbe1e7ed94f8537fc3424c6e6e7e
|
|
BLAKE2b-256 checksum How to use checksums |
1220b7bbfac01d295e83f46d6a73865676ea0eef3e30da21b7fcf060ad333d81
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logRelease files / vigily-0.1.0b1-py3-none-any.whl
| Download URL | vigily-0.1.0b1-py3-none-any.whl |
|---|---|
| Size | 55.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
60fe4d63f5bec7cf60a5a2e44c2693105f4bade3a7e44a701b549114af1ee299
|
|
BLAKE2b-256 checksum How to use checksums |
a5f8229865a22930bb3bcedffe9d1b64c3ad5fc16b7dc1ff324fe34171332d4e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logSecurity for vigily 0.1.0b1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for vigily-0.1.0b1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.