agent-eval
A utility for evaluating agents on a suite of Inspect-formatted evals, with the following primary benefits:
- Task suite specifications as config.
- Extracts the token usage of the agent from log files, and computes cost using
litellm. - Submits task suite results to a leaderboard, with submission metadata and easy upload to a HuggingFace repo for distribution of scores and logs.
Installation
To install from pypi, use pip install agent-eval.
For leaderboard extras, use pip install agent-eval[leaderboard].
Usage
Run evaluation suite
agenteval eval --config-path CONFIG_PATH --split SPLIT LOG_DIR
Evaluate an agent on the supplied eval suite configuration. Results are written to agenteval.json in the log directory.
See sample-config.yml for a sample configuration file.
For aggregation in a leaderboard, each task specifies a primary_metric as {scorer_name}/{metric_name}.
The scoring utils will look for a corresponding stderr metric,
by looking for another metric with the same scorer_name and with a metric_name containing the string "stderr".
Weighted Macro Averaging with Tags
Tasks can be grouped using tags for computing summary statistics. The tags support weighted macro averaging, allowing you to assign different weights to tasks within a tag group.
Tags are specified as simple strings on tasks. To adjust weights for specific tag-task combinations, use the macro_average_weight_adjustments field at the split level. Tasks not specified in the adjustments default to a weight of 1.0.
See sample-config.yml for an example of the tag and weight adjustment format.
Score results
agenteval score [OPTIONS] LOG_DIR
Compute scores for the results in agenteval.json and update the file with the computed scores.
Publish scores to leaderboard
agenteval lb publish [OPTIONS] LOG_DIR
Upload the scored results to HuggingFace datasets.
View leaderboard scores
agenteval lb view [OPTIONS]
View results from the leaderboard.
To save plots:
agenteval lb view --save-dir DIR [OPTIONS]
Administer the leaderboard
Prior to publishing scores, two HuggingFace datasets should be set up, one for full submissions and one for results files.
If you want to call load_dataset() on the results dataset (e.g., for populating a leaderboard), you probably want to explicitly tell HuggingFace about the schema and dataset structure (otherwise, HuggingFace may fail to propertly auto-convert to Parquet).
This is done by updating the configs attribute in the YAML metadata block at the top of the README.md file at the root of the results dataset (the metadata block is identified by lines with just --- above and below it).
This attribute should contain a list of configs, each of which specifies the schema (under the features key) and dataset structure (under the data_files key).
See sample-config-hf-readme-metadata.yml for a sample metadata block corresponding to sample-comfig.yml (note that the metadata references the raw schema data, which must be copied).
To facilitate initializing new configs, agenteval lb publish will automatically add this metadata if it is missing.
Development
See Development.md for development instructions.
1 maintainer
Metadata
1 maintainer
Release files for agent-eval 0.1.54
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| agent_eval-0.1.54.tar.gz | 52.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| agent_eval-0.1.54-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 98.1 kB
Release files / agent_eval-0.1.54.tar.gz
| Download URL | agent_eval-0.1.54.tar.gz |
|---|---|
| Size | 52.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4abcd86b2a92e171429dd270ba1fe201fc404bcbdc2c0a8fe1bea95eeaab6f0a
|
|
BLAKE2b-256 checksum How to use checksums |
625c4fe2c8f40a632f9d98ae0f4108a40df1cdec8a369300bf638045cb7de909
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.6
|
Release files / agent_eval-0.1.54-py3-none-any.whl
| Download URL | agent_eval-0.1.54-py3-none-any.whl |
|---|---|
| Size | 46.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0eb7065ad04ce858a0ca3e34c560cb9c880cc570cae0bf852f927179e08282aa
|
|
BLAKE2b-256 checksum How to use checksums |
e02d08faea0446c37090398d292b9b22ad31688e3b0f66f67475cb243927765b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.6
|
