VIPER

VIPER defines and verifies reproducible machine-learning experiments. Before a run, it fixes the dataset, source commit, stage specifications, environment, and reproducibility controls. After the run, it checks the recorded inputs, outputs, measurements, and trained model against that plan.

Install the viper-provenance distribution. Its Python package and command are both named viper.

Directory map

The focused model, verifier, and acceptance checks live in the repository test directory.

Protocol and verification flow

The protocol models divide requested state from realized state. A RunSpec and its ordered stage specs form the frozen run plan. Each completed stage publishes one ResolvedStageRef containing a resolved stage spec and all declared artifact files at one immutable snapshot.

RunSpec + ordered stage specs
              │
              ▼
    permitted runtime-state set
              │
              ▼
RunAttempt.resolved_stages[]
              │
              ▼
ResolvedStageRef.snapshot
├── resolved stage spec
└── exact files for every named artifact
              │
              ▼
          ResolvedRun
              │
              ▼
      verify_run_result()

The verifier starts from ResolvedRun.spec, verifies the exact RunSpec bytes, loads experiment and variant records, retrieves every stage spec, and checks the realized environment, command, inputs, artifacts, measurements, logs, and terminal estimator. Artifact loaders are selected by ArtifactSpec.loader from the exact Git commit recorded by RunSpec.source.

verify_benchmark_result() verifies a second successful attempt, its complete input lineage, estimator and prediction file parity, metric criteria, and result status. verify_promoted_artifact() verifies the selected producer run and any benchmark result required to authorize estimator promotion.

An evaluation measures one candidate. A benchmark standardizes that evaluation across candidates and requires a reproducible, threshold-qualified result. EvaluateSpec binds the candidate parameters, evaluation inputs, metrics, execution parameters, and outputs. BenchmarkSpec repeats the evaluation ID, dataset, splits, and metric IDs, then adds metric thresholds and a fixed execution count. The verifier requires the repeated values to match, which allows one benchmark to govern multiple candidate run plans.

Every stored input and produced artifact declares a data-use role: training, validation, evaluation, or benchmark. The verifier confirms stored-input roles against their producer artifacts, propagates same-run roles, prevents a stage from weakening an inherited restriction, and blocks evaluation or benchmark data from entering a training stage.

Public operations

• load_stage_spec(path) parses a DownloadSpec, BuildSpec, EmbedSpec, TrainSpec, or EvaluateSpec through the discriminated Spec union.
• load_resolved_stage(path) parses the corresponding realized record through ResolvedSpec.
• verify_run_result(resolved_run, policy=..., fetcher=...) verifies one terminal run and returns its connected run plan, successful resolved stages, and measurements.
• verify_benchmark_result(result, policy=..., fetcher=...) verifies the benchmark record, selected run, confirmation attempt, parity, and metric thresholds.
• verify_promoted_artifact(pointer, policy=..., fetcher=...) verifies a promoted artifact's producer lineage and benchmark authorization when required.
• capture_resume_state(...) captures optimizer state, main-process generator state, and the stateful DataLoader state at a training-stage boundary.
• restore_resume_state(...) restores those values before the next DataLoader iterator is created.
• save_resume_state(path, resume_state) and load_resume_state(path) write and safely load the reserved resume_state artifact.
• A custom fetcher receives a GitFileRef or HuggingFaceFileRef and returns bytes. Omitting it uses the package Git and Hugging Face retrieval functions.
• VerificationPolicy lists the exact source repositories whose artifact loader code may execute. Verification fails before loader retrieval when the run source is absent from that list.
• freeze_run_plan(repository_root, draft) validates and canonicalizes each stage spec, records its exact hash and byte count, and writes the sibling stage and run spec.yaml files.
• execute_stage_process(...) verifies the frozen stage-spec bytes, invokes the canonical command, and records every produced artifact file.
• preflight_plan(repository_root, run_spec_path) checks the complete plan on the active host and returns every applicable named check.
• run(repository_root, run_spec_path) executes every stage, publishes immutable stage results, writes the terminal resolved.yaml, and verifies the completed run.
• plan_diff(...) verifies and compares two RunSpecs and every stage spec they identify.
• lineage(...) verifies one terminal run and returns its directed upstream provenance graph.

Current local execution

Freeze the plan, inspect it, then execute it from the project repository:

viper freeze-run <draft.yaml>
viper preflight <run-spec-path>
viper run <run-spec-path>

RunSpec.source.commit identifies the project source, environment lockfile, metric implementations, and artifact loaders. The Git revision containing the run spec identifies the frozen plan files. VIPER checks both revisions before execution.

The local runner creates an exclusive attempt workspace, applies the run-wide reproducibility controls, invokes each stage through the VIPER runtime bootstrap, materializes declared inputs, publishes immutable results, and runs the complete verifier before returning success.

Python execution interface

Project code declares stage callables with VIPER decorators and executes them through ordinary Python:

import viper


@viper.train_stage(parameter_model=TrainParameters)
def train(context: viper.StageContext[TrainParameters]) -> None:
    ...


if __name__ == "__main__":
    viper.run(train)

python train.py --run <run-spec> --stage train

The installed viper run <run-spec> command will execute a complete plan through the same application coordinator. Local and GCE execution use the same interfaces. The user invokes VIPER inside the selected host; VIPER records and verifies that host's realized environment.

The approved mechanics live in the stage-invocation, process-startup, and cloud-execution contracts.

Validation

Run the fast development gate from the repository root after activating the mantra Conda environment described in the development guide:

make check

The command runs Ruff, formatting, Pyright, and the unit and contract tests. The integration gate exercises process, runner, CLI, resume, and durable-attempt boundaries. The release gate adds the complete generated-project acceptance case:

make check-integration
make check-release

Pytest still honors direct file selection. For example, python -m pytest tests/test_runner_acceptance.py -q runs that complete module.

Current boundaries

• LocalEnvironmentSpec and GCEEnvironmentSpec use the same in-place runner. GCE execution records the immutable provisioning image, machine type, CPU, CUDA backend, lockfile, and Python environment.
• Every internal stage binds its versioned JSON parameters to an exact project-owned Pydantic class. VIPER validates the class and values during plan freezing, preflight, and execution.
• Evaluation reserves the logical artifact name predictions. The project selects its file or bundle format and declares the exact loader path.
• Data-use roles are assigned by the project when source artifacts enter the provenance graph. VIPER verifies their propagation and permitted stage flows. The project assigns the scientific role when the artifact enters the graph.
• Each experiment metric records its role, parameters, and exact repository-relative implementation path.
• VIPER accepts any user source-tree layout. Stage scripts, metric implementations, and artifact loaders are selected by exact repository-relative paths and fixed by RunSpec.source.
• Artifact loaders execute Python from the Git commit named by RunSpec.source. Verification therefore accepts only run sources trusted to execute in the verifier process.
• The runner publishes successful, failed, cancelled, and preempted attempts, preserves completed-stage evidence, and retries the same frozen plan through viper.run(stage_callable), viper run, or viper retry.