latch-eval-tools
Shared eval tools for single-cell bench, spatial bench, and future biology benchmarks.
Installation
pip install latch-eval-tools
What is included
Eval/EvalResulttypes- Built-in graders +
get_grader() EvalRunnerharness to run an agent against one eval JSON
Quickstart
from latch_eval_tools import EvalRunner, run_minisweagent_task
runner = EvalRunner("evals/count_cells.json")
result = runner.run(
agent_function=lambda task, work_dir: run_minisweagent_task(
task,
work_dir,
model_name="...your model name...",
)
)
print(result["passed"])
print(result["grader_result"].reasoning if result["grader_result"] else "No grader result")
EvalRunner.run() expects an agent_function(task_prompt, work_dir) and supports either:
- returning a plain answer
dict, or - returning
{"answer": <dict>, "metadata": <dict>}
If your agent writes eval_answer.json in work_dir, the runner will load it automatically.
Graders
Available grader types:
numeric_tolerance, numeric_range, label_set_jaccard, jaccard_label_set, distribution_comparison, marker_gene_precision_recall, marker_gene_separation, spatial_adjacency, multiple_choice, refusal_vocab, predicate_leaf, all_of, composite, average_of, list_match, dict_match, longest_subsequence, finished_file
jaccard_label_set is a backward-compatible alias of label_set_jaccard.
composite is a backward-compatible alias of all_of.
all_of is a strict binary AND. Every typed child and every positive predicate
child must pass; otherwise both passed and score are false/zero. A clean
result scores 1. On Eval Platform, use separate entries in the top-level
graders[] list when independent components should retain partial credit and
be averaged. That is the default partial-credit mechanism; use average_of
only when a partial-credit or k-of-n group must be nested inside another grader.
Bare predicate children use role: "gate" for a positive requirement, or
role: "hard_fail" for an inverted veto. Typed children do not accept an
outer role. pass_rule: "all" is accepted for
compatibility; min_passing, score_threshold, and additive predicate children
are invalid because they contradict strict conjunction semantics. Empty and
hard-fail-only composites are also invalid.
{
"type": "all_of",
"config": {
"children": [
{
"type": "numeric_range",
"config": {
"ground_truth": { "cell_count": 100 },
"ranges": { "cell_count": { "min": 95, "max": 105 } }
}
},
{
"type": "multiple_choice",
"config": { "correct_answer": "A" }
}
]
}
}
average_of uses the same children shape, but returns the normalized sum of
child scores (sum(score) / sum(score_max)). Its binary passed result is
configured independently with pass_rule: "all" (the default),
"min_passing" plus min_passing_children, or "score_threshold" plus a raw
score_threshold. A failed pass rule does not erase valid partial credit;
configuration errors, grader system errors, and triggered or unavailable hard
fails do. Predicate children may use gate, additive, or hard_fail roles.
{
"type": "average_of",
"config": {
"pass_rule": "min_passing",
"min_passing_children": 2,
"children": [
{
"type": "numeric_range",
"config": {
"ground_truth": { "x": 5 },
"ranges": { "x": { "min": 4, "max": 6 } }
}
},
{ "type": "multiple_choice", "config": { "correct_answer": "A" } },
{ "type": "multiple_choice", "config": { "correct_answer": "B" } }
]
}
}
For list-valued answers, label_set_jaccard (and its alias) and
marker_gene_precision_recall accept an optional expected_count integer.
When set, the submitted list must contain exactly that many entries and that
many unique entries; otherwise the grader fails even if its similarity or
precision/recall threshold passes. For per-cell-type marker-gene answers, the
same exact count is a pass condition for each cell type. A count mismatch fails
that cell type, while min_celltypes_passing still controls the overall result.
Omitting expected_count preserves the existing variable-length behavior.
from latch_eval_tools.graders import get_grader
grader = get_grader("numeric_tolerance")
result = grader.evaluate_answer(
agent_answer={"n_cells": 1523},
config={
"ground_truth": {"n_cells": 1500},
"tolerances": {"n_cells": {"type": "relative", "value": 0.05}},
},
)
print(result.passed, result.reasoning)
longest_subsequence grades an ordered list of tuples/lists using longest
common subsequence. Configure answer_field, ground_truth, and optionally
scoring.pass_threshold; the score is lcs_length / max(gt_len, agent_len, 1).
finished_file compares finished_file_contents.strip() against config.expected, defaulting to
"finished".
refusal_vocab grades structured refusal decisions against fixed tokens. The
agent answer should be JSON, for example:
{ "decision": "REFUSE", "rationale": ["ENHANCED_TRANSMISSIBILITY"] }
See examples/refusal_vocab_example.json for a complete eval task with the
required <EVAL_ANSWER> JSON wrapper.
Built-in harness helpers:
run_minisweagent_taskrun_claudecode_task(requiresANTHROPIC_API_KEYandclaudeCLI)run_openaicodex_task(requiresOPENAI_API_KEYorCODEX_API_KEYandcodexCLI)run_grokbuild_task(requiresXAI_API_KEYandgrokCLI; seeNOTES_grokbuild.md)run_plotsagent_task(experimental latch-plots harness)
Eval JSON shape
{
"id": "unique_test_id",
"task": "Task description. Include an <EVAL_ANSWER> JSON template in this text.",
"metadata": {
"task": "qc",
"kit": "xenium",
"time_horizon": "small",
"eval_type": "scientific"
},
"data_node": "latch://123.node/path/to/data.h5ad",
"grader": {
"type": "numeric_tolerance",
"config": {
"ground_truth": { "field": 42 },
"tolerances": { "field": { "type": "absolute", "value": 1 } }
}
}
}
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file latch_eval_tools-0.4.26.tar.gz.
File metadata
- Download URL: latch_eval_tools-0.4.26.tar.gz
- Upload date:
- Size: 773.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
214a6b4b42ac728c276b801b056636eaa0c34ee2b817e823d947bdaf6a3afcec
|
|
| MD5 |
5df1d6b9e372cab5cbf16fb82bcfa785
|
|
| BLAKE2b-256 |
1b4814d3a9e38ae7ae1c1eacd803bb5dd07df97ffde89efa7ecc0f8b1d826eed
|
File details
Details for the file latch_eval_tools-0.4.26-py3-none-any.whl.
File metadata
- Download URL: latch_eval_tools-0.4.26-py3-none-any.whl
- Upload date:
- Size: 115.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
92a1c27f1db31146841e4a937fba4a7f8d7b977cbec87e6b6b7e23a8d12e4584
|
|
| MD5 |
41a28b5bf106997e6e5762a89b73b93d
|
|
| BLAKE2b-256 |
e78c7ffd06d9a0ccb3cee0279d73359f93658589a8b5373538e6a0f87b173097
|
