EvalForge Maestro
Evaluate. Compare. Verify.
"EvalForge Maestro is an independent evaluation and regression verification engine — never an LLM runtime, RAG indexer, or agent framework."
EvalForge Maestro provides deterministic evaluation metrics, retrieval/RAG diagnostics, agent trace verification, OpenTelemetry span ingestion, statistical multi-trial cohort comparison, model-based judge calibration, and CI/CD quality gating behind a strict Pydantic v2 schema contract (evalforge-maestro CLI and evalforge_maestro Python SDK).
Key Architecture & Guarantees
- Strict Domain Isolation: Zero coupling to document chunkers, runtime RAG stores, or agent orchestration loops. Target systems plug in via thin protocol adapters (
CallableAdapter,RAGAdapter, lazyDocpipeAdapter,AgentAdapter) or completely offline (ReplayAdapter,ingest_otel_spans,load_otel_executions). - Cold-Start Purity: Importing
evalforge_maestroor invokingevalforge-maestro --helpnever loads heavy or runtime-specific modules (litellm,docpipe_maestro,numpy,scipy,opentelemetry,opentelemetry_sdk,sqlite3,http.server) unless explicitly requested. - Deterministic & Immutable Artifacts: Canonical JSON hashing (
SHA-256) of datasets and redacted configurations paired with atomic0o700/0o600run records under.evalforge/runs/and append-onlymanifest.jsonl. - Defence-in-Depth Security: Pre-network
--local-onlyendpoint validation (loopback-only enforcement andLITELLM_LOCAL_MODEL_COST_MAP), opt-in--block-piiegress scanning, prompt-injection XML/nonce delimiting in model-based judges, and automatic credential/API-key/Bearer/PII masking ([REDACTED]) across logs, traces, resolved configs, and reports. - Separation of Concerns: Every run and report strictly separates Evaluation (metric outcomes + retrieval-vs-answer/trace decoupling diagnostics), Regression (baseline cohort deltas, $p$-values, sliding OLS trends, anomaly flags, drift warnings), and Acceptance / Quality Gate (policy pass/fail verdict and POSIX exit code).
How EvalForge Maestro Works (Visual Guide)
EvalForge Maestro follows a strict, deterministic six-stage pipeline that separates executing your AI system from scoring quality, checking regressions, and enforcing release gates:
flowchart TD
S1["1. Define Golden Dataset (JSONL / JSON / Docpipe Fixture)"]
S2A["Live Target: CallableAdapter / RAGAdapter / DocpipeAdapter / AgentAdapter"]
S2B["Offline Replay: ReplayAdapter (Captured JSONL or OpenTelemetry Spans)"]
S3["3. Pre-Network Security Preflight (--local-only / --block-pii / Secret Scrubbing)"]
S4["4. Bounded EvaluationOrchestrator + 4 Metric Families"]
S5["5. Three-Way Engine: Evaluation | Regression (vs Baseline / Cohort) | QualityGate"]
S6["6. Immutable Artifacts (.evalforge/runs/*.json + manifest.jsonl)"]
S7["Optional Layers: SQLiteRunIndex (index/query) | HTTP Service (serve)"]
S1 --> S2A
S1 --> S2B
S2A --> S3
S2B --> S3
S3 --> S4
S4 --> S5
S5 --> S6
S6 --> S7
Step-by-Step Pipeline Walkthrough
| Step | What Happens & Why It Matters | Concrete Components & Commands |
|---|---|---|
| 1. Define | Stream versioned test cases (EvaluationCase: input, reference, expected_context, expected_tool_calls, criteria.mandatory) and lock a canonical SHA-256 dataset_fingerprint so baseline comparisons never silently drift when test cases change. |
DatasetEngine, load_docpipe_fixture, --dataset |
| 2. Adapt / Ingest | Capture what your system actually produced (ExecutionResult: final output, retrieved_chunks, tool_calls, trace, tokens, latency). Run live against Python/RAG/Agent/Docpipe code or completely offline from pre-recorded executions or OTel spans. |
CallableAdapter, RAGAdapter, DocpipeAdapter, AgentAdapter, ReplayAdapter, ingest_otel_spans |
| 3. Security Preflight | Before any non-mock network call is attempted, --local-only rejects non-loopback judge/embedding endpoints and --block-pii scans payloads for emails, tokens, and credentials (exiting cleanly with POSIX 4). |
validate_local_only_target, validate_pii_egress, redact_secrets |
| 4. Orchestrate & Score | EvaluationOrchestrator runs cases under bounded concurrency (--max-concurrency), checkpoints per-case progress atomically, truncates oversized persisted traces after scoring (--max-context-chars), and evaluates up to 4 metric families simultaneously. |
Deterministic text/JSON, Retrieval IR (100% branch), Agent Trace (100% branch), LLM Judges & Dense Embeddings |
| 5. Three-Way Evaluate & Gate | Never collapses metrics into a single opaque average. Produces (a) raw Evaluation scores + decoupling diagnostics, (b) Regression deltas, Student's/Welch's $t$-test $p$-values (--repeats), OLS trend slopes, & anomaly flags (>0.3 spread), and (c) deterministic Quality Gate verdicts (PASS / FAIL / WARNING / INCONCLUSIVE). |
compare_runs, compare_run_cohorts, analyze_trend, QualityGate, --strict |
| 6. Persist, Index & Serve | Writes immutable 0o600 JSON RunRecord artifacts + append-only manifest.jsonl and pinned baselines.json. Optionally builds a rebuildable SQLite index (SQLiteRunIndex) for fast queries or exposes everything behind Bearer-authenticated HTTP (ServiceApplication / serve). |
LocalRunStore, SQLiteRunIndex, evalforge-maestro index / query / serve |
Inside the Metric Families & Three-Way Diagnosis
Rather than hiding failures behind a macro average, EvalForge Maestro isolates where a RAG or agent pipeline broke:
flowchart LR
EX["Captured ExecutionResult (output + retrieved_chunks + tool_calls/trace)"]
M1["Deterministic Text & Schema (exact_match, normalized_match, json_schema_validation)"]
M2["Retrieval IR Ranking (precision_at_k, recall_at_k, mrr, ndcg_at_k, hit_rate_at_k)"]
M3["Agent Tool & Trace (tool_selection, argument_correctness, execution_sequence, efficiency, policy, recovery)"]
M4["Semantic & Judge Scoring (--mock offline OR local/remote LiteLLM: groundedness, relevance, correctness, embeddings)"]
DEC{"Three-Way & Decoupling Diagnostics"}
G1["PASS / FAIL / WARNING / INCONCLUSIVE (POSIX Exit 0..4)"]
EX --> M1
EX --> M2
EX --> M3
EX --> M4
M1 --> DEC
M2 --> DEC
M3 --> DEC
M4 --> DEC
DEC -->|"Retrieval OK + Generation Failed -> Fix Prompt / LLM"| G1
DEC -->|"Retrieval Failed -> Fix Chunking / Hybrid / Reranker"| G1
DEC -->|"Trace Violation / Policy Breach -> Fix Agent Guardrails"| G1
Calibrating Small & Local Judges First: Before letting a local model (such as Ollama/vLLM) gate pull requests, run
evalforge-maestro calibrate --samples ... --metric groundedness --min-agreement 80.0to measureexact_agreement_rate, directional bias (too_generous_ratevs.too_harsh_rate),mean_score_delta, andrecommended_pass_threshold(docs/calibration-and-ci.md).
Minimal Core vs. Lazy Optional Extras
Core installation keeps dependencies minimal and cold-start instant; optional runtimes are only imported when invoked:
flowchart LR
CORE["evalforge-maestro Core (pydantic + jsonschema only): CLI, Datasets, Deterministic/IR/Agent Metrics, Statistical t-tests, OTel Ingestion, SQLite Index, HTTP Service"]
E1["[docpipe-maestro]: DocpipeAdapter (FTS5 hybrid + cross-encoder rerank + answer_detailed)"]
E2["[judges]: LiteLLM Gateway (Groundedness, Answer/Context Relevance, Custom Rubric, Calibration)"]
E3["[embeddings]: numpy + LiteLLM (EmbeddingSimilarityMetric, ContextEmbeddingSimilarityMetric)"]
CORE -.->|"lazy import -> MissingExtraError if absent"| E1
CORE -.->|"lazy import -> MissingExtraError if absent"| E2
CORE -.->|"lazy import -> MissingExtraError if absent"| E3
Installation
Install core evaluation, deterministic/retrieval/agent metrics, offline replay, and zero-dependency OpenTelemetry ingestion:
pip install evalforge-maestro
Optional capability extras (lazy-loaded on demand; never required at import time):
# Docpipe Maestro retrieval and RAG target adapter
pip install "evalforge-maestro[docpipe-maestro]"
# LiteLLM-backed model-as-judge evaluation and calibration
pip install "evalforge-maestro[judges]"
# Dense embedding similarity metrics (numpy + litellm)
pip install "evalforge-maestro[embeddings]"
For contributor development using uv:
uv sync --extra dev
CLI Quickstart Reference
Both evalforge-maestro and evalforge_maestro console commands expose eleven subcommands: run, ingest-otel, index, query, compare, trend, calibrate, baseline, report, config, and serve.
1. Offline CI / Local Regression Evaluation
evalforge-maestro run \
--config examples/ci/evalforge.toml \
--mock \
--local-only \
--strict \
--json \
--json-out report.json \
--markdown-out report.md
2. Multi-Trial Statistical Cohorts & Regression Gating
Evaluate stochastic LLM targets across N repeatable seeded runs and enforce Student's / Welch's $t$-test regression gates:
evalforge-maestro run \
--config examples/ci/evalforge.toml \
--repeats 3 \
--seed 10 \
--baseline release-v1 \
--fail-on-significant-regression \
--json
3. OpenTelemetry Trace Ingestion & Replay (ingest-otel / run --otel-traces)
Evaluate OTLP JSON envelopes, flat span arrays, or JSONL span streams directly (or normalize them into replayable ExecutionResult JSONL artifacts) with zero OTel SDK dependencies:
# Normalize an OTel span export into replayable ExecutionResult JSONL
evalforge-maestro ingest-otel examples/ci/otel-traces-sample.json \
--dataset examples/ci/golden-sample.jsonl \
--output replays.jsonl \
--system-version agent-v2.4.0 \
--json
# Evaluate OTel traces directly with agent + retrieval metrics and strict quality gating
evalforge-maestro run \
--config examples/ci/evalforge-otel.toml \
--strict \
--max-context-chars 4000 \
--json
See the full OpenTelemetry Trace Ingestion Guide for GenAI / OpenInference / Traceloop attribute precedence, tool call deduplication, clock-skew clamping, token anti-double-counting, and correlation rules.
4. Dense Embedding Similarity Evaluation
evalforge-maestro run \
--config examples/ci/evalforge.toml \
--metric embedding_similarity \
--metric context_embedding_similarity:aggregation=mean \
--mock \
--local-only \
--json
5. Baseline Pinning, Pairwise / Cohort Comparison, Trends & Calibration
# Pin a trusted baseline without mutating immutable run artifacts
evalforge-maestro baseline set <RUN_ID> --name release-v1 --json
# Compare single candidate or multi-trial candidate cohort against baseline(s)
evalforge-maestro compare <CANDIDATE_RUN_ID> --baseline release-v1 --strict --json
evalforge-maestro compare <CAND_R01> <CAND_R02> \
--baseline release-v1 --baseline-run <BASE_R02> \
--fail-on-significant-regression --json
# Inspect sliding-window OLS metric trend slopes
evalforge-maestro trend --window 5 --json
# Calibrate a judge metric against labeled human samples with minimum agreement gate
evalforge-maestro calibrate \
--samples examples/ci/calibration-samples.jsonl \
--metric groundedness \
--min-agreement 80.0 \
--mock \
--local-only \
--json
# Render Three-Way Markdown/JSON diagnostic reports from stored artifacts
evalforge-maestro report --run <RUN_ID> --baseline release-v1 > diagnostic-report.md
# Audit effective resolved configuration with provenance tiers (secrets always masked)
evalforge-maestro config validate --config evalforge.toml --json
evalforge-maestro config show --resolved
6. Local SQLite Query Index (index & query)
Build or sync an optional derived SQLite query index (.evalforge/index.sqlite3) from canonical JSON artifacts and query runs or failing cases cleanly—deleting or corrupting the DB never loses data and rebuilds atomically:
# Inspect sync parity or atomically rebuild from .evalforge/runs/*.json + manifest.jsonl
evalforge-maestro index status --storage-dir .evalforge --json
evalforge-maestro index rebuild --storage-dir .evalforge --json
# Query historical runs and triage mandatory/failing cases
evalforge-maestro query runs --storage-dir .evalforge --dataset-id support-qa --metric exact_match --min-score 0.8 --json
evalforge-maestro query cases --storage-dir .evalforge --status FAIL --mandatory-only --json
See the Rebuildable SQLite Query Index Guide for full recovery guarantees, normalized DDL tables, and filter references.
7. Optional Remote Evaluation HTTP Service (serve)
Serve canonical storage queries, SQLite index lifecycle, OTel trace ingestion, offline/trace evaluation, comparison gates, and report rendering over a typed stdlib HTTP/1.1 JSON API (safe loopback default 127.0.0.1:8787, local_only=True, pre-bind remote auth guard, and constant-time Bearer authentication):
# Local loopback server in offline mock mode (defaults to local_only=True)
evalforge-maestro serve --storage-dir .evalforge --mock --log-level INFO
# Non-loopback host requires --auth-token or EVALFORGE_SERVICE_TOKEN (else exits 4 pre-bind)
EVALFORGE_SERVICE_TOKEN="strong-secret-token" evalforge-maestro serve \
--host 0.0.0.0 \
--port 8787 \
--storage-dir .evalforge \
--allow-remote-judges \
--block-pii \
--log-level INFO
See the Remote Evaluation Service Guide for security guarantees, complete route/schema tables, curl examples, and in-process ServiceApplication embedding.
Configuration Precedence, TOML Schema & Exit Codes
5-Tier Precedence Order
- CLI Flags (
--dataset,--otel-traces,--strict, etc.) - OS Environment Variables (
EVALFORGE_*, e.g.EVALFORGE_JUDGE_API_KEY,EVALFORGE_OTEL_TRACES,EVALFORGE_OTEL_CASE_ID_ATTR) - Opt-in Environment File (
--env-file <path>) - Configuration File (
--config <path>or non-recursive./evalforge.tomlin CWD) - Built-in Safe Defaults (
mock_mode=false,storage_dir=".evalforge",max_concurrency=4, strict security guards)
TOML configuration validates strictly (extra="forbid") across [run], [target], [otel], [metrics], [judge], [embedding], [security], [gate], and [logging] ([judge] and [embedding] credentials/endpoints remain strictly isolated).
Output Streams
--json: Emits strictly one schema-validated, secret-redacted JSON payload onstdout; diagnostics, warnings, and--log-json/--log-levellogs write exclusively tostderr.- Default (non-
--json): Writes structured Markdown summary reports tostdout(suppressible via--quiet).--json-outand--markdown-outwrite report files concurrently.
POSIX Exit Codes
| Exit Code | Meaning | Typical Causes |
|---|---|---|
0 |
Success | Gate PASS (or WARNING/INCONCLUSIVE when --strict is disabled), diagnostic calibrate, ingest-otel, baseline, report, or config. |
1 |
Quality Gate Failure | Metric threshold breach, mandatory case failure, regression drop violation, significant regression, or --strict escalating WARNING/INCONCLUSIVE. |
2 |
Input / Config / Alignment Error | Invalid CLI flags, unknown evalforge.toml keys, missing file, corrupt JSON/JSONL, replay alignment mismatch, conflicting --otel-traces + --executions/--target, or unresolved OTel case_id. |
3 |
Runtime / Extra Failure | Adapter or judge execution fault under --fail-fast, zero cases completed, or missing optional extra (MissingExtraError). |
4 |
Security / Policy Violation | Pre-network block triggered by --local-only non-loopback endpoint or --block-pii PII egress detection. |
Python SDK Quickstarts
RAG & Deterministic Evaluation
from evalforge_maestro import (
DatasetEngine,
EvaluationOrchestrator,
PrecisionAtKMetric,
RAGAdapter,
RecallAtKMetric,
exact_match,
)
dataset = DatasetEngine(
[
{
"id": "q1",
"input": "What is the capital of France?",
"reference": "Paris",
"expected_context": ["doc-paris"],
}
]
)
adapter = RAGAdapter(
lambda q: {
"answer": "Paris",
"chunks": [{"id": "doc-paris", "text": "Paris is the capital of France."}],
}
)
run = EvaluationOrchestrator().run(
dataset,
adapter,
[exact_match, PrecisionAtKMetric(k=1), RecallAtKMetric(k=1)],
)
print(run.aggregates["recall_at_k"].mean_score)
OpenTelemetry Trace Ingestion & Agent Replay
from evalforge_maestro import (
ArgumentCorrectnessMetric,
DatasetEngine,
EvaluationOrchestrator,
OtelExtractorConfig,
ReplayAdapter,
ToolSelectionMetric,
ingest_otel_spans,
load_otel_executions,
)
# 1. Normalize in-memory OTLP dicts, flat span lists, or duck-typed SDK ReadableSpan objects
executions = ingest_otel_spans(otlp_payload)
# 2. Or load from disk (OTLP JSON / array JSON / JSONL) into ReplayAdapter for orchestrator evaluation
dataset = DatasetEngine("examples/ci/golden-sample.jsonl")
adapter = ReplayAdapter.from_otel_file(
"examples/ci/otel-traces-sample.json",
extractors=OtelExtractorConfig(match_dataset_by_input=True),
dataset=dataset,
)
run = EvaluationOrchestrator().run(
dataset,
adapter,
[ToolSelectionMetric(), ArgumentCorrectnessMetric()],
)
print(run.aggregates["tool_selection"].mean_score)
Rebuildable SQLite Run & Case Querying
from evalforge_maestro import CaseQueryFilter, MetricStatus, RunQueryFilter, SQLiteRunIndex
index = SQLiteRunIndex(".evalforge")
runs = index.query_runs(
RunQueryFilter(dataset_id="support-qa", metric_id="exact_match", min_mean_score=0.8)
)
failed = index.query_cases(CaseQueryFilter(status=MetricStatus.FAIL, mandatory_only=True))
In-Process HTTP Service Dispatcher (ServiceApplication)
import json
from evalforge_maestro import ServiceApplication, ServiceConfig
app = ServiceApplication(
ServiceConfig(storage_dir=".evalforge", local_only=True, mock_mode=True),
auth_token="service-secret",
)
status, headers, body = app.handle_request(
"GET",
"/health",
headers={"Authorization": "Bearer service-secret"},
)
assert status == 200 and json.loads(body.decode("utf-8"))["status"] == "ok"
Documentation & CI Templates
- Remote Evaluation Service Guide (
docs/remote-service.md): Strict scope boundary, security posture (pre-bind auth policy,hmac.compare_digest,local_only,--block-pii,413size limits), complete 13-endpoint API reference,curlworkflows, and in-processServiceApplicationembedding. - Consumer CI & Small-Judge Calibration Guide (
docs/calibration-and-ci.md): Downstreamdocpipe-maestrofts5_hybrid+ cross-encoder rerank CI templates (evalforge-hybrid-rerank.toml), statistical cohort regression gating, andcalibrate --min-agreementthreshold calibration for local/small judges. - OpenTelemetry Trace Ingestion Guide: Complete specification of supported OTel formats, GenAI/OpenInference/Traceloop attribute mapping, span ordering, tool deduplication, token anti-double-counting, clock-skew handling, case correlation, and CLI/SDK examples.
- Rebuildable SQLite Query Index Guide: Canonical-JSON single-source-of-truth guarantees, atomic rebuild/sync mechanics, corruption recovery, SQLite DDL schema, CLI
index/queryflags, and Python SDK workflows. - Consumer CI Templates (
examples/ci/README.md): Copy-ready GitHub Actions workflowevalforge-ci.yml, offline replay configevalforge.toml, livedocpipe-maestroconfigevalforge-docpipe.toml, OTel trace replay configevalforge-otel.toml, hybrid/rerank CI configevalforge-hybrid-rerank.toml, and human judge calibration samplescalibration-samples.jsonl. - Product Intent & Boundaries (
docs/intent.md): Domain scope, architectural invariants, security guarantees, and non-goals. - Post-MVP Roadmap (
docs/future-features.md): Complete milestone inventory and downstream adoption status.
Implemented Milestones (M1–M6 + Post-M6 Extensions)
- M1 — Evaluation Foundation (
001-evaluation-foundation): Single Pydantic v2 schema source (evalforge_maestro.types), lazy JSONL streamingDatasetEngine,CallableAdapter, deterministic text/JSON Schema metrics (exact_match,normalized_match,json_schema_validation), bounded asyncEvaluationOrchestrator, and atomic0o700/0o600.evalforge/storage. - M2 — Retrieval & RAG Evaluation (
002-retrieval-rag-evaluation): Protocol-basedRAGAdapter, lazyDocpipeAdapter([docpipe-maestro]), dependency-free fixture importer, and versioned IR metrics (precision_at_k,recall_at_k,mrr,ndcg_at_k,hit_rate_at_k) with AST formula verification (100%branch coverage). - M3 — Experiment & Regression Engine (
003-experiment-regression-engine): Pinned baseline aliases, pairwise aggregate/per-case comparisons, sliding-window OLS trend slopes (analyze_trend), anti-masking/disagreement anomalies (> 0.3divergence), provenance drift alerts, and deterministicQualityGate(100%branch coverage). - M4 — Model-Based Judge & Calibration Engine (
004-judge-calibration-engine): Lazy LiteLLM gateway ([judges]), pre-network--local-only&LITELLM_LOCAL_MODEL_COST_MAPair-gap safety, randomized XML/nonce prompt-injection delimiting, offline--mockscoring, context-grouped bounded async batching, and human calibration (calibrate_judge/CalibrationReport). - M5 — CLI, Reporting & CI Integration (
005-cli-reporting-ci): Dual CLI entrypoints (evalforge-maestro/evalforge_maestro), offlineReplayAdapter, opt-in--block-piiegress guard, library-pureevalforge_maestro.reporting(ReportDocumentJSON & Three-Way Markdown separating Evaluation, Regression, and Quality Gate + retrieval-vs-answer decoupling diagnostics), and copy-ready CI workflow templates. - M6 — Agent Trace Evaluation (
006-agent-trace-evaluation): Backward-compatible typed trace/tool contracts (ExpectedToolCall,ToolCallRecord,AgentTraceStep), six100%-branch-covered deterministic agent metrics (tool_selection,argument_correctness,execution_sequence,tool_efficiency,policy_compliance,failure_recovery), dependency-freeAgentAdapter+ structural OpenAI/Anthropic trace normalizers, CLI--target-type agentsupport, field-level post-scoringmax_context_charstrace truncation, and decoupled trace-vs-answer diagnostics. - Statistical & Embedding Metrics (
007-statistical-embedding-metrics): Lazy[embeddings]extra (numpy+litellm) poweringEmbeddingSimilarityMetricandContextEmbeddingSimilarityMetricwith strict[judge]/[embedding]isolation, SHA-256--mockfallback, pre-network--local-only/--block-piichecks, pure-stdlib Student's $t$ / Welch's $t$ / one-sample $t$ confidence intervals and $p$-values (compare_runs,compare_run_cohorts,build_pooled_cohort_run),--repeatsimmutable cohorts, andQualityGate.evaluate_cohort(fail_on_significant_regression). - OpenTelemetry Execution Trace Ingestion (
008-otel-trace-ingestion): Dependency-free ingestion (ingest_otel_spans,load_otel_executions,ReplayAdapter.from_otel_file,OtelExtractorConfig,OtelTimestampUnit) normalizing OTLP JSON, flat span arrays, JSONL spans, and duck-typed SDKReadableSpanobjects across GenAI/OpenInference/Traceloop conventions into secret-scrubbed canonicalExecutionResulttraces, plus[otel]/[run].otel_tracesconfig precedence,evalforge-maestro run --otel-traces, andevalforge-maestro ingest-otel. - Rebuildable Local SQLite Query Index (
009-sqlite-query-index): Zero-dependencySQLiteRunIndex(evalforge_maestro.storage) backed atomically by canonical.evalforge/runs/*.json,manifest.jsonl, andbaselines.jsonwith0o700/0o600permissions,PRAGMA journal_mode=DELETE, transparent deletion/corruption recovery, parameterizedquery_runs/query_cases/metric_history, and CLIevalforge-maestro index&querysubcommands. - Optional Remote Evaluation Service & Consumer CI Guides (
010-remote-eval-service): Library-pureServiceApplication(evalforge_maestro.service), lazy stdlibThreadingHTTPServer(create_http_server,serve_http), and CLIevalforge-maestro serveenforcing pre-bind remote auth policies (validate_service_bind_policy), constant-time Bearer auth,local_only=Truedefaults,HTTP 413body limits, secret scrubbing, and zerohttp.servercold-start leakage—paired with verified hybrid/rerank (evalforge-hybrid-rerank.toml) & small-judge calibration (calibration-samples.jsonl) CI fixtures and comprehensive guides (docs/remote-service.md,docs/calibration-and-ci.md).
CI/CD & PyPI Trusted Publishing Setup
- CI Pipeline (
.github/workflows/ci.yml): Enforcesuv lock --check,ruff format --check,ruff check, BanditSchecks,mypy --strict,>=85%global branch coverage,100%branch coverage onevalforge_maestro.metricsandevalforge_maestro.regression.gate,@pytest.mark.gate_determinism/@pytest.mark.gate_immutability,pip-audit,cz check, cross-OS testing (ubuntu-latest,macos-latest,windows-latest),uv build+twine check --strict, clean-venv wheel/sdist installs + CLI cold-start smoke checks, and--no-devMissingExtraErrorisolation. - Release Pipeline (
.github/workflows/release.yml): Triggered onv*tags; re-runs all verification gates, verifieshatch-vcswheel version matches the Git tag, publishes via OIDC Trusted Publishing (id-token: write) toTestPyPI, validates clean wheel + CLI entrypoint installation fromTestPyPI, promotes toPyPI, and creates the GitHub Release.- One-time repository prerequisite before pushing the first
v*tag: configure GitHub environments namedtestpypiandpypiand register matching OIDC Trusted Publishers ontest.pypi.organdpypi.orgtargeting.github/workflows/release.yml.
- One-time repository prerequisite before pushing the first
Metadata
Release files for evalforge-maestro 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| evalforge_maestro-0.3.0.tar.gz | 1.2 MB | With provenance | Details for evalforge_maestro-0.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| evalforge_maestro-0.3.0-py3-none-any.whl | Python 3 | none | any | Details for evalforge_maestro-0.3.0-py3-none-any.whl |
Total release size: 1.4 MB
Release files / evalforge_maestro-0.3.0.tar.gz
| Download URL | evalforge_maestro-0.3.0.tar.gz |
|---|---|
| Size | 1.2 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3459d330f6f00e69c96d921089fac5e6ebc156cf8004ab48f797c4e744564973
|
|
BLAKE2b-256 checksum How to use checksums |
7ef830e3046a31dc2d71aadeda5d6488de38027ece7258d170ef3f7bf075d019
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logRelease files / evalforge_maestro-0.3.0-py3-none-any.whl
| Download URL | evalforge_maestro-0.3.0-py3-none-any.whl |
|---|---|
| Size | 204.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3ff1b3bc54bab6f5f8bcb42bab1406e2f0e6d1e03b675b327411899f7d8ef8ec
|
|
BLAKE2b-256 checksum How to use checksums |
1bdced7ae63aaf23172dacd661f19170abf7c0264b1f03dd9ca1bbf86be55b4f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logSecurity for evalforge-maestro 0.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for evalforge_maestro-0.3.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.