Skip to main content

EvalForge Maestro

Evaluate. Compare. Verify.

"EvalForge Maestro is an independent evaluation and regression verification engine — never an LLM runtime, RAG indexer, or agent framework."

Installation

pip install evalforge-maestro
# With optional docpipe-maestro integration:
pip install "evalforge-maestro[docpipe]"
# With optional LiteLLM model-based judges:
pip install "evalforge-maestro[judges]"

For contributor development using uv:

uv sync --extra dev

Quickstart (CLI)

Both evalforge-maestro and evalforge_maestro console commands are installed:

# Run offline deterministic + mock judge evaluation cleanly in CI or locally
evalforge-maestro run \
  --config examples/ci/evalforge.toml \
  --mock \
  --local-only \
  --strict \
  --json \
  --json-out report.json \
  --markdown-out report.md

# Pin a trusted baseline without mutating immutable run records
evalforge-maestro baseline set <RUN_ID> --name release-v1 --json

# Compare a candidate run against a pinned baseline
evalforge-maestro compare <CANDIDATE_RUN_ID> --baseline release-v1 --strict --json

# Inspect multi-run sliding OLS trends
evalforge-maestro trend --window 5 --json

# Calibrate a judge against human ground-truth samples
evalforge-maestro calibrate --samples cal-samples.jsonl --metric groundedness --mock --local-only --json

# Export Markdown/JSON diagnostic reports from stored artifacts
evalforge-maestro report --run <RUN_ID> --baseline release-v1 > diagnostic-report.md

# Audit effective resolved config across precedence tiers (secrets always masked)
evalforge-maestro config show --resolved

Configuration Precedence & Streams

  • Precedence: CLI flags > OS environment (EVALFORGE_*) > opt-in --env-file <path> > non-recursive CWD evalforge.toml > Built-in defaults.
  • Streams: --json emits strictly one schema-validated JSON document on stdout; diagnostics, warnings, and --log-json / --log-level logs write strictly to stderr. Default non---json runs write Markdown summaries to stdout (suppressible via --quiet).
  • POSIX Exit Codes:
    • 0: Gate PASS (or WARNING/INCONCLUSIVE when --strict is disabled), diagnostic calibrate, or management commands.
    • 1: Gate FAIL (mandatory failure, threshold breach, excessive regression, or --strict escalating WARNING/INCONCLUSIVE).
    • 2: CLI argument, evalforge.toml / --env-file configuration, replay alignment, missing run/baseline, or dataset/schema validation error.
    • 3: Adapter, judge, or missing-extra runtime failure (--fail-fast, MissingExtraError, or zero cases completed).
    • 4: Security or locality policy violation (--local-only endpoint block or --block-pii pre-network egress block).

Quickstart (Python SDK)

from evalforge_maestro import (
    DatasetEngine,
    EvaluationOrchestrator,
    PrecisionAtKMetric,
    RAGAdapter,
    RecallAtKMetric,
    exact_match,
)

dataset = DatasetEngine(
    [
        {
            "id": "q1",
            "input": "What is the capital of France?",
            "reference": "Paris",
            "expected_context": ["doc-paris"],
        }
    ]
)

adapter = RAGAdapter(
    lambda q: {
        "answer": "Paris",
        "chunks": [{"id": "doc-paris", "text": "Paris is the capital of France."}],
    }
)

run = EvaluationOrchestrator().run(
    dataset,
    adapter,
    [exact_match, PrecisionAtKMetric(k=1), RecallAtKMetric(k=1)],
)
print(run.aggregates["recall_at_k"].mean_score)

GitHub Actions CI Template

Copy examples/ci/evalforge-ci.yml and examples/ci/evalforge.toml into your project to gate pull requests offline (--mock --local-only) and publish Markdown summaries directly to $GITHUB_STEP_SUMMARY.

Implemented MVP Milestones (M1–M5)

  • M1 — Evaluation Foundation (001-evaluation-foundation): Single Pydantic v2 schema source (evalforge_maestro.types), lazy JSONL streaming DatasetEngine, CallableAdapter, deterministic text/JSON Schema metrics (exact_match, normalized_match, json_schema_validation), bounded async EvaluationOrchestrator, and atomic 0o700/0o600 .evalforge/ storage.
  • M2 — Retrieval & RAG Evaluation (002-retrieval-rag-evaluation): Protocol-based RAGAdapter, lazy DocpipeAdapter ([docpipe]), dependency-free fixture importer, and versioned IR metrics (precision_at_k, recall_at_k, mrr, ndcg_at_k, hit_rate_at_k) with AST formula verification.
  • M3 — Experiment & Regression Engine (003-experiment-regression-engine): Pinned baseline aliases, pairwise aggregate/per-case comparisons, sliding-window OLS trend slopes (analyze_trend), anti-masking/disagreement anomalies (> 0.3 divergence), provenance drift alerts, and deterministic QualityGate.
  • M4 — Model-Based Judge & Calibration Engine (004-judge-calibration-engine): Lazy LiteLLM gateway ([judges]), pre-network --local-only & LITELLM_LOCAL_MODEL_COST_MAP air-gap safety, randomized XML/nonce prompt-injection delimiting, offline --mock scoring, context-grouped bounded async batching, and human calibration (calibrate_judge / CalibrationReport).
  • M5 — CLI, Reporting & CI Integration (005-cli-reporting-ci): Dual CLI entrypoints (evalforge-maestro / evalforge_maestro) with seven subcommands (run, compare, trend, calibrate, baseline, report, config), offline ReplayAdapter, opt-in --block-pii egress guard, library-pure evalforge_maestro.reporting (ReportDocument JSON & Three-Way Markdown separating Evaluation, Regression, and Quality Gate + retrieval-vs-answer decoupling diagnostics), and copy-ready CI workflow templates.

CI/CD & PyPI Trusted Publishing Setup

  • CI Pipeline (.github/workflows/ci.yml): Enforces uv lock --check, ruff format --check, ruff check, Bandit S checks, mypy --strict, >=85% global branch coverage, 100% branch coverage on evalforge_maestro.metrics and evalforge_maestro.regression.gate, @pytest.mark.gate_determinism / @pytest.mark.gate_immutability, pip-audit, cz check, cross-OS testing (ubuntu-latest, macos-latest, windows-latest), uv build + twine check --strict, clean-venv wheel/sdist installs + CLI cold-start smoke checks, and --no-dev MissingExtraError isolation.
  • Release Pipeline (.github/workflows/release.yml): Triggered on v* tags; re-runs all verification gates, verifies hatch-vcs wheel version matches the Git tag, publishes via OIDC Trusted Publishing (id-token: write) to TestPyPI, validates clean wheel + CLI entrypoint installation from TestPyPI, promotes to PyPI, and creates the GitHub Release.
    • One-time repository prerequisite before pushing the first v* tag: configure GitHub environments named testpypi and pypi and register matching OIDC Trusted Publishers on test.pypi.org and pypi.org targeting .github/workflows/release.yml.

Metadata

Release files for evalforge-maestro 0.1.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for evalforge-maestro 0.1.3
File Size Uploaded Provenance Actions
evalforge_maestro-0.1.3.tar.gz 925.0 kB With provenance Details for evalforge_maestro-0.1.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for evalforge-maestro 0.1.3
File Interpreter ABI Platform Actions
evalforge_maestro-0.1.3-py3-none-any.whl Python 3 none any Details for evalforge_maestro-0.1.3-py3-none-any.whl

Total release size: 1.0 MB

Release files / evalforge_maestro-0.1.3.tar.gz

Download URL evalforge_maestro-0.1.3.tar.gz
Size 925.0 kB
Tags Source
SHA-256 checksum
How to use checksums
85b71ea88db1699473b114b21d9860f3b12318d1dce264851ae8c408175e29cc
BLAKE2b-256 checksum
How to use checksums
9b58c295699f1f7cec44e4e043a356f29dc0d1a557d5dc57cc4b75329386319b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Release files / evalforge_maestro-0.1.3-py3-none-any.whl

Download URL evalforge_maestro-0.1.3-py3-none-any.whl
Size 121.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
bf99fa5ad3fc0fd6b932ab96f53ca604a3eecca0c931d199c4e524a6119a2023
BLAKE2b-256 checksum
How to use checksums
9834c995a8208addb674f4b12e4dd2b775ac27ad6321f8d42a5d1bb2c70c5821
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Security for evalforge-maestro 0.1.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for evalforge_maestro-0.1.3.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page