Skip to main content

EvalForge Maestro

Evaluate. Compare. Verify.

"EvalForge Maestro is an independent evaluation and regression verification engine — never an LLM runtime, RAG indexer, or agent framework."

Installation

pip install evalforge-maestro
# With optional docpipe-maestro integration:
pip install "evalforge-maestro[docpipe]"
# With optional LiteLLM model-based judges:
pip install "evalforge-maestro[judges]"

For contributor development using uv:

uv sync --extra dev

Quickstart (CLI)

Both evalforge-maestro and evalforge_maestro console commands are installed:

# Run offline deterministic + mock judge evaluation cleanly in CI or locally
evalforge-maestro run \
  --config examples/ci/evalforge.toml \
  --mock \
  --local-only \
  --strict \
  --json \
  --json-out report.json \
  --markdown-out report.md

# Pin a trusted baseline without mutating immutable run records
evalforge-maestro baseline set <RUN_ID> --name release-v1 --json

# Compare a candidate run against a pinned baseline
evalforge-maestro compare <CANDIDATE_RUN_ID> --baseline release-v1 --strict --json

# Inspect multi-run sliding OLS trends
evalforge-maestro trend --window 5 --json

# Calibrate a judge against human ground-truth samples
evalforge-maestro calibrate --samples cal-samples.jsonl --metric groundedness --mock --local-only --json

# Export Markdown/JSON diagnostic reports from stored artifacts
evalforge-maestro report --run <RUN_ID> --baseline release-v1 > diagnostic-report.md

# Audit effective resolved config across precedence tiers (secrets always masked)
evalforge-maestro config show --resolved

Configuration Precedence & Streams

  • Precedence: CLI flags > OS environment (EVALFORGE_*) > opt-in --env-file <path> > non-recursive CWD evalforge.toml > Built-in defaults.
  • Streams: --json emits strictly one schema-validated JSON document on stdout; diagnostics, warnings, and --log-json / --log-level logs write strictly to stderr. Default non---json runs write Markdown summaries to stdout (suppressible via --quiet).
  • POSIX Exit Codes:
    • 0: Gate PASS (or WARNING/INCONCLUSIVE when --strict is disabled), diagnostic calibrate, or management commands.
    • 1: Gate FAIL (mandatory failure, threshold breach, excessive regression, or --strict escalating WARNING/INCONCLUSIVE).
    • 2: CLI argument, evalforge.toml / --env-file configuration, replay alignment, missing run/baseline, or dataset/schema validation error.
    • 3: Adapter, judge, or missing-extra runtime failure (--fail-fast, MissingExtraError, or zero cases completed).
    • 4: Security or locality policy violation (--local-only endpoint block or --block-pii pre-network egress block).

Quickstart (Python SDK)

from evalforge_maestro import (
    DatasetEngine,
    EvaluationOrchestrator,
    PrecisionAtKMetric,
    RAGAdapter,
    RecallAtKMetric,
    exact_match,
)

dataset = DatasetEngine(
    [
        {
            "id": "q1",
            "input": "What is the capital of France?",
            "reference": "Paris",
            "expected_context": ["doc-paris"],
        }
    ]
)

adapter = RAGAdapter(
    lambda q: {
        "answer": "Paris",
        "chunks": [{"id": "doc-paris", "text": "Paris is the capital of France."}],
    }
)

run = EvaluationOrchestrator().run(
    dataset,
    adapter,
    [exact_match, PrecisionAtKMetric(k=1), RecallAtKMetric(k=1)],
)
print(run.aggregates["recall_at_k"].mean_score)

GitHub Actions CI Template

Copy examples/ci/evalforge-ci.yml and examples/ci/evalforge.toml (for offline --executions replay) or examples/ci/evalforge-docpipe.toml (for [docpipe] retrieval gating) into your project to gate pull requests offline (--mock --local-only) and publish Markdown summaries directly to $GITHUB_STEP_SUMMARY. See examples/ci/README.md for consumer setup details.

Implemented Milestones (M1–M6)

  • M1 — Evaluation Foundation (001-evaluation-foundation): Single Pydantic v2 schema source (evalforge_maestro.types), lazy JSONL streaming DatasetEngine, CallableAdapter, deterministic text/JSON Schema metrics (exact_match, normalized_match, json_schema_validation), bounded async EvaluationOrchestrator, and atomic 0o700/0o600 .evalforge/ storage.
  • M2 — Retrieval & RAG Evaluation (002-retrieval-rag-evaluation): Protocol-based RAGAdapter, lazy DocpipeAdapter ([docpipe]), dependency-free fixture importer, and versioned IR metrics (precision_at_k, recall_at_k, mrr, ndcg_at_k, hit_rate_at_k) with AST formula verification.
  • M3 — Experiment & Regression Engine (003-experiment-regression-engine): Pinned baseline aliases, pairwise aggregate/per-case comparisons, sliding-window OLS trend slopes (analyze_trend), anti-masking/disagreement anomalies (> 0.3 divergence), provenance drift alerts, and deterministic QualityGate.
  • M4 — Model-Based Judge & Calibration Engine (004-judge-calibration-engine): Lazy LiteLLM gateway ([judges]), pre-network --local-only & LITELLM_LOCAL_MODEL_COST_MAP air-gap safety, randomized XML/nonce prompt-injection delimiting, offline --mock scoring, context-grouped bounded async batching, and human calibration (calibrate_judge / CalibrationReport).
  • M5 — CLI, Reporting & CI Integration (005-cli-reporting-ci): Dual CLI entrypoints (evalforge-maestro / evalforge_maestro) with seven subcommands (run, compare, trend, calibrate, baseline, report, config), offline ReplayAdapter, opt-in --block-pii egress guard, library-pure evalforge_maestro.reporting (ReportDocument JSON & Three-Way Markdown separating Evaluation, Regression, and Quality Gate + retrieval-vs-answer decoupling diagnostics), and copy-ready CI workflow templates.
  • M6 — Agent Trace Evaluation (006-agent-trace-evaluation): Backward-compatible typed trace/tool contracts (ExpectedToolCall, ToolCallRecord, AgentTraceStep), six 100%-branch-covered deterministic agent metrics (tool_selection, argument_correctness, execution_sequence, tool_efficiency, policy_compliance, failure_recovery), dependency-free AgentAdapter + structural OpenAI/Anthropic trace normalizers, CLI --target-type agent support, field-level post-scoring max_context_chars trace truncation, and decoupled trace-vs-answer diagnostics.

CI/CD & PyPI Trusted Publishing Setup

  • CI Pipeline (.github/workflows/ci.yml): Enforces uv lock --check, ruff format --check, ruff check, Bandit S checks, mypy --strict, >=85% global branch coverage, 100% branch coverage on evalforge_maestro.metrics and evalforge_maestro.regression.gate, @pytest.mark.gate_determinism / @pytest.mark.gate_immutability, pip-audit, cz check, cross-OS testing (ubuntu-latest, macos-latest, windows-latest), uv build + twine check --strict, clean-venv wheel/sdist installs + CLI cold-start smoke checks, and --no-dev MissingExtraError isolation.
  • Release Pipeline (.github/workflows/release.yml): Triggered on v* tags; re-runs all verification gates, verifies hatch-vcs wheel version matches the Git tag, publishes via OIDC Trusted Publishing (id-token: write) to TestPyPI, validates clean wheel + CLI entrypoint installation from TestPyPI, promotes to PyPI, and creates the GitHub Release.
    • One-time repository prerequisite before pushing the first v* tag: configure GitHub environments named testpypi and pypi and register matching OIDC Trusted Publishers on test.pypi.org and pypi.org targeting .github/workflows/release.yml.

Metadata

Release files for evalforge-maestro 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for evalforge-maestro 0.2.0
File Size Uploaded Provenance Actions
evalforge_maestro-0.2.0.tar.gz 981.3 kB With provenance Details for evalforge_maestro-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for evalforge-maestro 0.2.0
File Interpreter ABI Platform Actions
evalforge_maestro-0.2.0-py3-none-any.whl Python 3 none any Details for evalforge_maestro-0.2.0-py3-none-any.whl

Total release size: 1.1 MB

Release files / evalforge_maestro-0.2.0.tar.gz

Download URL evalforge_maestro-0.2.0.tar.gz
Size 981.3 kB
Tags Source
SHA-256 checksum
How to use checksums
4570668726bde64219326c7c63f5737dd59c1dfe3832c32e895f266bdd8d94b7
BLAKE2b-256 checksum
How to use checksums
f8c5573968adec0a79e59634df7a933270bc7acf607f2e093168e0de67ddc209
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Release files / evalforge_maestro-0.2.0-py3-none-any.whl

Download URL evalforge_maestro-0.2.0-py3-none-any.whl
Size 140.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
867be4d96c4829a5ef98d197fc7b71344660d790ce8e937ffd1ac75f2f1eecba
BLAKE2b-256 checksum
How to use checksums
20f7acb16de5fd10b8d75500cd267d44503be59062118526a274c20fa4ebcead
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

Security for evalforge-maestro 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for evalforge_maestro-0.2.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page