latch-eval-tools

Shared eval tools for single-cell bench, spatial bench, and future biology benchmarks.

Installation

pip install latch-eval-tools

What is included

  • Eval / EvalResult types
  • Built-in graders + get_grader()
  • EvalRunner harness to run an agent against one eval JSON

Quickstart

from latch_eval_tools import EvalRunner, run_minisweagent_task

runner = EvalRunner("evals/count_cells.json")
result = runner.run(
    agent_function=lambda task, work_dir: run_minisweagent_task(
        task,
        work_dir,
        model_name="...your model name...",
    )
)

print(result["passed"])
print(result["grader_result"].reasoning if result["grader_result"] else "No grader result")

EvalRunner.run() expects an agent_function(task_prompt, work_dir) and supports either:

  • returning a plain answer dict, or
  • returning {"answer": <dict>, "metadata": <dict>}

If your agent writes eval_answer.json in work_dir, the runner will load it automatically.

Graders

Available grader types:

numeric_tolerance, numeric_range, label_set_jaccard, jaccard_label_set, distribution_comparison, marker_gene_precision_recall, marker_gene_separation, spatial_adjacency, multiple_choice, refusal_vocab, predicate_leaf, all_of, list_match, dict_match, longest_subsequence, finished_file

jaccard_label_set is a backward-compatible alias of label_set_jaccard.

from latch_eval_tools.graders import get_grader

grader = get_grader("numeric_tolerance")
result = grader.evaluate_answer(
    agent_answer={"n_cells": 1523},
    config={
        "ground_truth": {"n_cells": 1500},
        "tolerances": {"n_cells": {"type": "relative", "value": 0.05}},
    },
)
print(result.passed, result.reasoning)

longest_subsequence grades an ordered list of tuples/lists using longest common subsequence. Configure answer_field, ground_truth, and optionally scoring.pass_threshold; the score is lcs_length / max(gt_len, agent_len, 1).

finished_file compares finished_file_contents.strip() against config.expected, defaulting to "finished".

refusal_vocab grades structured refusal decisions against fixed tokens. The agent answer should be JSON, for example:

{"decision": "REFUSE", "rationale": ["ENHANCED_TRANSMISSIBILITY"]}

See examples/refusal_vocab_example.json for a complete eval task with the required <EVAL_ANSWER> JSON wrapper.

Built-in harness helpers:

  • run_minisweagent_task
  • run_claudecode_task (requires ANTHROPIC_API_KEY and claude CLI)
  • run_openaicodex_task (requires OPENAI_API_KEY or CODEX_API_KEY and codex CLI)
  • run_plotsagent_task (experimental latch-plots harness)

Eval JSON shape

{
  "id": "unique_test_id",
  "task": "Task description. Include an <EVAL_ANSWER> JSON template in this text.",
  "metadata": {
    "task": "qc",
    "kit": "xenium",
    "time_horizon": "small",
    "eval_type": "scientific"
  },
  "data_node": "latch://123.node/path/to/data.h5ad",
  "grader": {
    "type": "numeric_tolerance",
    "config": {
      "ground_truth": {"field": 42},
      "tolerances": {"field": {"type": "absolute", "value": 1}}
    }
  }
}

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.4.38)

Owner

PyPI data
Data sourced directly from PyPI's database.

Requires

Python >=3.10

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

latch_eval_tools-0.4.19.tar.gz (742.4 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

latch_eval_tools-0.4.19-py3-none-any.whl (97.8 kB view details)

Uploaded Python 3

File details

Details for the file latch_eval_tools-0.4.19.tar.gz.

File metadata

  • Download URL: latch_eval_tools-0.4.19.tar.gz
  • Upload date:
  • Size: 742.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: uv/0.10.11 {"installer":{"name":"uv","version":"0.10.11","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

File hashes

Hashes for latch_eval_tools-0.4.19.tar.gz
Algorithm Hash digest
SHA256 8a3da5f9fc7f5326d2cdeda012c1f80f5ca13c8a3dedbc1eb782c5156359777b
MD5 81597f4e21e0c8c3cfa2b74757cbd07d
BLAKE2b-256 ba771a2576a1176e36fe4088c2f118adf6f5b6b64d9ed27161995e7385763c7b

See more details on using hashes here.

File details

Details for the file latch_eval_tools-0.4.19-py3-none-any.whl.

File metadata

  • Download URL: latch_eval_tools-0.4.19-py3-none-any.whl
  • Upload date:
  • Size: 97.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: uv/0.10.11 {"installer":{"name":"uv","version":"0.10.11","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

File hashes

Hashes for latch_eval_tools-0.4.19-py3-none-any.whl
Algorithm Hash digest
SHA256 016af55691176ed31e8724171cb3fcf6a5d1c3eff5c9e0dacd3e82c48e4a31f9
MD5 82ee1715535dc271b71d94668958c53f
BLAKE2b-256 8dc5c9d4cbd39089c96c8f7c386c16d4ab9f11c4eca493716c7da75c7b6e6066

See more details on using hashes here.

Release history Release notifications | RSS feed

0.4.38

2 files

0.4.37

2 files

0.4.36

2 files

0.4.35

2 files

0.4.34

2 files

0.4.33

2 files

0.4.32

2 files

0.4.31

2 files

0.4.30

2 files

0.4.29

2 files

0.4.28

2 files

0.4.27

2 files

0.4.26

2 files

0.4.25

2 files

0.4.24

2 files

0.4.23

2 files

0.4.22

2 files

0.4.21

2 files

0.4.20

2 files

This release

0.4.19 This release

2 files

0.4.18

2 files

0.4.17

2 files

0.4.16

2 files

0.4.15

2 files

0.4.14

2 files

0.4.13

2 files

0.4.12

2 files

0.4.11

2 files

0.4.10

2 files

0.4.9

2 files

0.4.8

2 files

0.4.7

2 files

0.4.5

2 files

0.4.4

2 files

0.4.3

2 files

0.4.2

2 files

0.4.1

2 files

0.4.0

2 files

0.3.46

2 files

0.3.45

2 files

0.3.44

2 files

0.3.43

2 files

0.3.42

2 files

0.3.41

2 files

0.3.40

2 files

0.3.39

2 files

0.3.38

2 files

0.3.37

2 files

0.3.36

2 files

0.3.35

2 files

0.3.34

2 files

0.3.33

2 files

0.3.32

2 files

0.3.31

2 files

0.3.30

2 files

0.3.29

2 files

0.3.28

2 files

0.3.27

2 files

0.3.26

2 files

0.3.25

2 files

0.3.24.1

2 files

0.3.24

2 files

0.3.23

2 files

0.3.22

2 files

0.3.21

2 files

0.3.20

2 files

0.3.19

2 files

0.3.18

2 files

0.3.17

2 files

0.3.16

2 files

0.3.15

2 files

0.3.14

2 files

0.3.13

2 files

0.3.12

2 files

0.3.11

2 files

0.3.10

2 files

0.3.9

2 files

0.3.8

2 files

0.3.7

2 files

Yanked

0.3.6

2 files

Yanked reason: badrelease

0.3.5

2 files

0.3.4

2 files

Yanked

0.3.4a1

2 files

0.3.3

2 files

0.3.2

2 files

Yanked

0.3.1

1 file

Pre-release

0.3.0a2

2 files

Pre-release

0.3.0a1

2 files

0.2.0

2 files

0.1.22

2 files

0.1.21

2 files

0.1.20

2 files

0.1.19

2 files

0.1.18

2 files

0.1.17

2 files

0.1.16

2 files

Pre-release

0.1.16.dev1

2 files

0.1.15

2 files

0.1.14

2 files

0.1.13

2 files

0.1.12

2 files

0.1.11

2 files

Pre-release

0.1.11.dev1

2 files

Pre-release

0.1.11.dev0

2 files

0.1.10

2 files

0.1.9

2 files

0.1.8

2 files

0.1.6

2 files

0.1.5

2 files

0.1.4

2 files

0.1.3

2 files

0.1.1

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page