library-eval
Scores MAS runs with quality metrics and LLM-as-judge prompts. It wraps the Metrics Computation Engine (MCE), an open-source set of LLM-as-judge metrics from the AGNTCY telemetry-hub project, for MAS traces.
Use it when you want a number for "did the system reach the goal" or "was the answer grounded" for one trace or a whole experiment.
Quick start
uv pip install -e library-eval # from the repository root; needs Python 3.11+
mas-lab eval --list-metrics
mas-lab eval path/to/events.jsonl --metric goal_success_rate --metric groundedness
Batch scoring over a benchmark output tree:
mas-lab eval path/to/experiment/ --metric goal_success_rate --recursive
mas-lab eval SOURCE flags:
| Flag | Meaning |
|---|---|
--metric |
Metric to compute; repeatable. |
--model |
Judge model name (default azure/gpt-4o). |
--api-base |
LLM proxy base URL (default https://api.openai.com/v1). |
--api-key-env |
Environment variable holding the API key (default OPENAI_API_KEY). |
--json |
Print results as JSON. |
--list-metrics |
List available metrics and exit; SOURCE is then optional. |
--recursive |
Score every trace under the SOURCE directory. |
The package depends on lab/components/core, lab/components/bench and mas-runtime from this repository, plus the public MCE packages:
metrics-computation-engine: MCE core.mce-metrics-plugin: quality metrics (GoalSuccessRate, Groundedness, ResponseCompleteness, ...).deepeval: continuous-score implementations ofanswer_relevancyandgoal_success_rate.
What the library provides
- Library code (
mas.library.eval.*): MCE integration, session entity construction, metric computation, and theeval_metricplugin type. - A CLI component (
mas.library.eval.cli) that registersmas-lab evalthrough themas.lab.cli.componentsentry point:
[project.entry-points."mas.lab.cli.components"]
eval = "mas.library.eval.cli:EvalCliComponent"
The eval_mce pipeline step, which scores a run into metrics.json, ships in library-lab (mas.library.lab.steps.eval.mce) and calls into this library. mas.library.eval.steps holds an older CamelCase-metric step class that is not registered.
src/mas/library/eval/
+-- evaluator.py EvalProvider + get_provider("mce")
+-- providers/ mce.py: built-in MCE provider (stock + registered metrics);
| trip_planner_gt.py, adversarial.py: sample ground-truth / adversarial providers
+-- mce/
| +-- catalog.py one catalog: METRIC_MAP + METRIC_REGISTRY
| +-- runner.py native-trace MCE/deepeval compute
| +-- registry_api.py SessionEntity CamelCase API
+-- metrics/ EvalMetric, plugins, prompt_metrics, stock wrappers, toy example
+-- steps/ older CamelCase-metric step class (not registered)
+-- cli/ mas-lab eval
Score runs in a benchmark
Attach the eval_mce step to a run. Metric ids are snake_case and the judge model comes from the experiment (see Tutorial 10):
# pipeline attached with `mas-lab benchmark run ... --pipeline run:post:eval.yaml`
apiVersion: mas/v1
kind: Pipeline
spec:
steps:
- name: eval
type: eval_mce
per_run: true
in: trace
out: metrics
config:
metrics: [goal_success_rate, groundedness, response_completeness]
Metric names
The CLI, the eval_mce step metrics: and the metrics.json keys use snake_case ids. The class names used by the SessionEntity API (compute_session_metrics) are CamelCase.
| Id (CLI and YAML) | Class name |
|---|---|
goal_success_rate |
GoalSuccessRate |
answer_relevancy |
AnswerRelevancy |
mas-lab eval --list-metrics lists the ids. The 11 stock session metrics are answer_relevancy, component_conflict_rate, consistency, context_preservation, goal_success_rate, groundedness, information_retention, intent_recognition_accuracy, response_completeness, workflow_cohesion_index and workflow_efficiency. The rule-based core metrics (ToolError, ToolErrorRate, ToolUtilizationAccuracy, CyclesCount, AgentToAgentInteractions, AgentToToolInteractions) are available through the CamelCase API only.
Register your own metric (eval_metric plugin)
A library registers session metrics on the MCE eval provider through the eval_metric plugin type (EvalProvider.register_metric). eval_mce then runs any mix of stock MCE ids and registered ids into one metrics.json. Each metric declares the input it needs (unit / evidence). Stock MCE ids score MAS input and output, not the raw trajectory.
# library.yaml
types:
- eval_metric
plugins:
- type: eval_metric
name: toy_echo
module: mas.library.eval.metrics.toy
class: ToyEchoMetric
attributes:
provider: mce # default
# A family that yields many ids from one factory:
- type: eval_metric
name: my_family
factory: mas.library.example.metrics:build_metrics
attributes:
provider: mce
Rules:
- Duplicate metric ids raise at registration.
- Stock MCE session ids are
EvalMetrics onget_provider("mce")(one batch per run). - Plugins and inline
prompt_metricsrun through the samecompute_metricsengine. - Unknown ids error with the list of available ids.
- Metrics that share
batch_keyare computed once per run viacompute_batch. - A failing metric writes
{value: null, error: "..."}for that id only. - Optional
detailson each score is additive inmetrics.jsonschema v1.
List registered ids in the metrics: list of an eval_mce step, or split families across steps with distinct metrics_filename values (metrics.json, metrics_judge2.json). For a one-off LLM-as-judge question, add prompt_metrics: [{id, prompt, unit, evidence}] to the same step without a plugin; ids must not collide with a stock or registered metric. collect_metrics and metrics_to_dataframe accept a filename, a glob or a list, and copy details into a JSON column.
Programmatic use
import os
from mas.library.eval.mce import build_session_from_trace, compute_session_metrics
session_entity = build_session_from_trace("path/to/events.jsonl")
results = await compute_session_metrics( # async
session=session_entity,
metrics=["GoalSuccessRate", "Groundedness"],
llm_config={
"LLM_MODEL_NAME": "azure/gpt-4o",
"LLM_BASE_MODEL_URL": "https://api.openai.com/v1",
"LLM_API_KEY": os.environ["OPENAI_API_KEY"],
},
)
llm_config is a dict as above, or an LLMJudgeConfig with the same three fields:
from metrics_computation_engine.models.requests import LLMJudgeConfig
llm_config = LLMJudgeConfig(
LLM_MODEL_NAME="azure/gpt-4o",
LLM_BASE_MODEL_URL="https://api.openai.com/v1",
LLM_API_KEY=os.environ["OPENAI_API_KEY"],
)
See also
- Tutorial 10, Evaluation metrics: how stock MCE, prompts, plugins, judge infra and trajectory wrappers fit together.
- Tutorial 3, Run an experiment: first MCE scoring.
- Feature example (not a sample app): examples/mce/judge-override/
- summarization.md, MCE judge
- MCE documentation
- mce_metrics_plugin
Metadata
Release files for mas-lab-library-eval 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| mas_lab_library_eval-0.2.0.tar.gz | 54.5 kB | With provenance | Details for mas_lab_library_eval-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| mas_lab_library_eval-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for mas_lab_library_eval-0.2.0-py3-none-any.whl |
Total release size: 111.8 kB
Release files / mas_lab_library_eval-0.2.0.tar.gz
| Download URL | mas_lab_library_eval-0.2.0.tar.gz |
|---|---|
| Size | 54.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7ec5b84ad23e0df2c06eed6036fb15a6a5b99e437fe1947b69c8914eb48ce0ec
|
|
BLAKE2b-256 checksum How to use checksums |
8e83d6782acee465ffe508fadc8043adaf3f8094cf1248980409d43d271ab260
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.
Transparency logRelease files / mas_lab_library_eval-0.2.0-py3-none-any.whl
| Download URL | mas_lab_library_eval-0.2.0-py3-none-any.whl |
|---|---|
| Size | 57.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
79b6629df39c53924351e6b1b61e33c76fa829f3df7726efee49a19e6b4520fe
|
|
BLAKE2b-256 checksum How to use checksums |
25260eada0c7d2fdaab93ff70b001eaa101a2907c31d90955aef1357134e8168
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.
Transparency logSecurity for mas-lab-library-eval 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for mas_lab_library_eval-0.2.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.