EvalForge Maestro
Evaluate. Compare. Verify.
"EvalForge Maestro is an independent evaluation and regression verification engine — never an LLM runtime, RAG indexer, or agent framework."
Installation
pip install evalforge-maestro
# With optional docpipe-maestro integration:
pip install "evalforge-maestro[docpipe]"
# With optional LiteLLM model-based judges:
pip install "evalforge-maestro[judges]"
For contributor development using uv:
uv sync --extra dev
Quickstart (CLI)
Both evalforge-maestro and evalforge_maestro console commands are installed:
# Run offline deterministic + mock judge evaluation cleanly in CI or locally
evalforge-maestro run \
--config examples/ci/evalforge.toml \
--mock \
--local-only \
--strict \
--json \
--json-out report.json \
--markdown-out report.md
# Pin a trusted baseline without mutating immutable run records
evalforge-maestro baseline set <RUN_ID> --name release-v1 --json
# Compare a candidate run against a pinned baseline
evalforge-maestro compare <CANDIDATE_RUN_ID> --baseline release-v1 --strict --json
# Inspect multi-run sliding OLS trends
evalforge-maestro trend --window 5 --json
# Calibrate a judge against human ground-truth samples
evalforge-maestro calibrate --samples cal-samples.jsonl --metric groundedness --mock --local-only --json
# Export Markdown/JSON diagnostic reports from stored artifacts
evalforge-maestro report --run <RUN_ID> --baseline release-v1 > diagnostic-report.md
# Audit effective resolved config across precedence tiers (secrets always masked)
evalforge-maestro config show --resolved
Configuration Precedence & Streams
- Precedence:
CLI flags > OS environment (EVALFORGE_*) > opt-in --env-file <path> > non-recursive CWD evalforge.toml > Built-in defaults. - Streams:
--jsonemits strictly one schema-validated JSON document onstdout; diagnostics, warnings, and--log-json/--log-levellogs write strictly tostderr. Default non---jsonruns write Markdown summaries tostdout(suppressible via--quiet). - POSIX Exit Codes:
0: GatePASS(orWARNING/INCONCLUSIVEwhen--strictis disabled), diagnosticcalibrate, or management commands.1: GateFAIL(mandatory failure, threshold breach, excessive regression, or--strictescalatingWARNING/INCONCLUSIVE).2: CLI argument,evalforge.toml/--env-fileconfiguration, replay alignment, missing run/baseline, or dataset/schema validation error.3: Adapter, judge, or missing-extra runtime failure (--fail-fast,MissingExtraError, or zero cases completed).4: Security or locality policy violation (--local-onlyendpoint block or--block-piipre-network egress block).
Quickstart (Python SDK)
from evalforge_maestro import (
DatasetEngine,
EvaluationOrchestrator,
PrecisionAtKMetric,
RAGAdapter,
RecallAtKMetric,
exact_match,
)
dataset = DatasetEngine(
[
{
"id": "q1",
"input": "What is the capital of France?",
"reference": "Paris",
"expected_context": ["doc-paris"],
}
]
)
adapter = RAGAdapter(
lambda q: {
"answer": "Paris",
"chunks": [{"id": "doc-paris", "text": "Paris is the capital of France."}],
}
)
run = EvaluationOrchestrator().run(
dataset,
adapter,
[exact_match, PrecisionAtKMetric(k=1), RecallAtKMetric(k=1)],
)
print(run.aggregates["recall_at_k"].mean_score)
GitHub Actions CI Template
Copy examples/ci/evalforge-ci.yml and examples/ci/evalforge.toml into your project to gate pull requests offline (--mock --local-only) and publish Markdown summaries directly to $GITHUB_STEP_SUMMARY.
Implemented MVP Milestones (M1–M5)
- M1 — Evaluation Foundation (
001-evaluation-foundation): Single Pydantic v2 schema source (evalforge_maestro.types), lazy JSONL streamingDatasetEngine,CallableAdapter, deterministic text/JSON Schema metrics (exact_match,normalized_match,json_schema_validation), bounded asyncEvaluationOrchestrator, and atomic0o700/0o600.evalforge/storage. - M2 — Retrieval & RAG Evaluation (
002-retrieval-rag-evaluation): Protocol-basedRAGAdapter, lazyDocpipeAdapter([docpipe]), dependency-free fixture importer, and versioned IR metrics (precision_at_k,recall_at_k,mrr,ndcg_at_k,hit_rate_at_k) with AST formula verification. - M3 — Experiment & Regression Engine (
003-experiment-regression-engine): Pinned baseline aliases, pairwise aggregate/per-case comparisons, sliding-window OLS trend slopes (analyze_trend), anti-masking/disagreement anomalies (> 0.3divergence), provenance drift alerts, and deterministicQualityGate. - M4 — Model-Based Judge & Calibration Engine (
004-judge-calibration-engine): Lazy LiteLLM gateway ([judges]), pre-network--local-only&LITELLM_LOCAL_MODEL_COST_MAPair-gap safety, randomized XML/nonce prompt-injection delimiting, offline--mockscoring, context-grouped bounded async batching, and human calibration (calibrate_judge/CalibrationReport). - M5 — CLI, Reporting & CI Integration (
005-cli-reporting-ci): Dual CLI entrypoints (evalforge-maestro/evalforge_maestro) with seven subcommands (run,compare,trend,calibrate,baseline,report,config), offlineReplayAdapter, opt-in--block-piiegress guard, library-pureevalforge_maestro.reporting(ReportDocumentJSON & Three-Way Markdown separating Evaluation, Regression, and Quality Gate + retrieval-vs-answer decoupling diagnostics), and copy-ready CI workflow templates.
CI/CD & PyPI Trusted Publishing Setup
- CI Pipeline (
.github/workflows/ci.yml): Enforcesuv lock --check,ruff format --check,ruff check, BanditSchecks,mypy --strict,>=85%global branch coverage,100%branch coverage onevalforge_maestro.metricsandevalforge_maestro.regression.gate,@pytest.mark.gate_determinism/@pytest.mark.gate_immutability,pip-audit,cz check, cross-OS testing (ubuntu-latest,macos-latest,windows-latest),uv build+twine check --strict, clean-venv wheel/sdist installs + CLI cold-start smoke checks, and--no-devMissingExtraErrorisolation. - Release Pipeline (
.github/workflows/release.yml): Triggered onv*tags; re-runs all verification gates, verifieshatch-vcswheel version matches the Git tag, publishes via OIDC Trusted Publishing (id-token: write) toTestPyPI, validates clean wheel + CLI entrypoint installation fromTestPyPI, promotes toPyPI, and creates the GitHub Release.- One-time repository prerequisite before pushing the first
v*tag: configure GitHub environments namedtestpypiandpypiand register matching OIDC Trusted Publishers ontest.pypi.organdpypi.orgtargeting.github/workflows/release.yml.
- One-time repository prerequisite before pushing the first
Metadata
Release files for evalforge-maestro 0.1.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| evalforge_maestro-0.1.4.tar.gz | 928.2 kB | With provenance | Details for evalforge_maestro-0.1.4.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| evalforge_maestro-0.1.4-py3-none-any.whl | Python 3 | none | any | Details for evalforge_maestro-0.1.4-py3-none-any.whl |
Total release size: 1.1 MB
Release files / evalforge_maestro-0.1.4.tar.gz
| Download URL | evalforge_maestro-0.1.4.tar.gz |
|---|---|
| Size | 928.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fe266510355478de771c83d231f6ba45254b02fda6ae718641b4efb280231454
|
|
BLAKE2b-256 checksum How to use checksums |
997961525321662313938cbd8ad7580ebef7c16f5b2cfde5c18dc1a78f43f470
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logRelease files / evalforge_maestro-0.1.4-py3-none-any.whl
| Download URL | evalforge_maestro-0.1.4-py3-none-any.whl |
|---|---|
| Size | 123.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
dec491aa232a6e17c9037cc6d8b726206b1279073aa4cd7c49f83ba35aebf78f
|
|
BLAKE2b-256 checksum How to use checksums |
29a97ee45b7e5be959ccf7a9dc0a2bec6434fb907ced0e5b228b11977e79c964
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logSecurity for evalforge-maestro 0.1.4
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for evalforge_maestro-0.1.4.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 20, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.