latch-eval-tools

Shared eval tools for single-cell bench, spatial bench, and future biology benchmarks.

Installation

pip install latch-eval-tools

What is included

  • Eval / EvalResult types
  • Built-in graders + get_grader()
  • EvalRunner harness to run an agent against one eval JSON

Quickstart

from latch_eval_tools import EvalRunner, run_minisweagent_task

runner = EvalRunner("evals/count_cells.json")
result = runner.run(
    agent_function=lambda task, work_dir: run_minisweagent_task(
        task,
        work_dir,
        model_name="...your model name...",
    )
)

print(result["passed"])
print(result["grader_result"].reasoning if result["grader_result"] else "No grader result")

EvalRunner.run() expects an agent_function(task_prompt, work_dir) and supports either:

  • returning a plain answer dict, or
  • returning {"answer": <dict>, "metadata": <dict>}

If your agent writes eval_answer.json in work_dir, the runner will load it automatically.

Graders

Available grader types:

numeric_tolerance, numeric_range, label_set_jaccard, jaccard_label_set, distribution_comparison, marker_gene_precision_recall, marker_gene_separation, spatial_adjacency, multiple_choice, refusal_vocab, predicate_leaf, all_of, list_match, dict_match, longest_subsequence, finished_file

jaccard_label_set is a backward-compatible alias of label_set_jaccard.

from latch_eval_tools.graders import get_grader

grader = get_grader("numeric_tolerance")
result = grader.evaluate_answer(
    agent_answer={"n_cells": 1523},
    config={
        "ground_truth": {"n_cells": 1500},
        "tolerances": {"n_cells": {"type": "relative", "value": 0.05}},
    },
)
print(result.passed, result.reasoning)

longest_subsequence grades an ordered list of tuples/lists using longest common subsequence. Configure answer_field, ground_truth, and optionally scoring.pass_threshold; the score is lcs_length / max(gt_len, agent_len, 1).

finished_file compares finished_file_contents.strip() against config.expected, defaulting to "finished".

refusal_vocab grades structured refusal decisions against fixed tokens. The agent answer should be JSON, for example:

{"decision": "REFUSE", "rationale": ["ENHANCED_TRANSMISSIBILITY"]}

See examples/refusal_vocab_example.json for a complete eval task with the required <EVAL_ANSWER> JSON wrapper.

Built-in harness helpers:

  • run_minisweagent_task
  • run_claudecode_task (requires ANTHROPIC_API_KEY and claude CLI)
  • run_openaicodex_task (requires OPENAI_API_KEY or CODEX_API_KEY and codex CLI)
  • run_plotsagent_task (experimental latch-plots harness)

Eval JSON shape

{
  "id": "unique_test_id",
  "task": "Task description. Include an <EVAL_ANSWER> JSON template in this text.",
  "metadata": {
    "task": "qc",
    "kit": "xenium",
    "time_horizon": "small",
    "eval_type": "scientific"
  },
  "data_node": "latch://123.node/path/to/data.h5ad",
  "grader": {
    "type": "numeric_tolerance",
    "config": {
      "ground_truth": {"field": 42},
      "tolerances": {"field": {"type": "absolute", "value": 1}}
    }
  }
}

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.4.38)

Owner

PyPI data
Data sourced directly from PyPI's database.

Requires

Python >=3.10

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

latch_eval_tools-0.4.15.tar.gz (734.7 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

latch_eval_tools-0.4.15-py3-none-any.whl (95.2 kB view details)

Uploaded Python 3

File details

Details for the file latch_eval_tools-0.4.15.tar.gz.

File metadata

  • Download URL: latch_eval_tools-0.4.15.tar.gz
  • Upload date:
  • Size: 734.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: uv/0.11.8 {"installer":{"name":"uv","version":"0.11.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

File hashes

Hashes for latch_eval_tools-0.4.15.tar.gz
Algorithm Hash digest
SHA256 6c86705c4a90908c5f6cea4fdee646d3eab437ca6e28dd19862ba994767a5900
MD5 168ccb519ec1f7be3cf88b22e31b9501
BLAKE2b-256 44dfd0e8bd04f85c8ee938db6586538c84e33535f2b60cbc4cdb1de049bb2465

See more details on using hashes here.

File details

Details for the file latch_eval_tools-0.4.15-py3-none-any.whl.

File metadata

  • Download URL: latch_eval_tools-0.4.15-py3-none-any.whl
  • Upload date:
  • Size: 95.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: uv/0.11.8 {"installer":{"name":"uv","version":"0.11.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

File hashes

Hashes for latch_eval_tools-0.4.15-py3-none-any.whl
Algorithm Hash digest
SHA256 fa440948c5bd0826cb10d399713e9b702cc37347269b5e07c070ff344b6db8bc
MD5 c21d723a2e59fb591ac126d43939669b
BLAKE2b-256 00c02f07fafb49bdd76d81b402ad9a39ed142432adc02f8a1ba495d54afaf3aa

See more details on using hashes here.

Release history Release notifications | RSS feed

0.4.38

2 files

0.4.37

2 files

0.4.36

2 files

0.4.35

2 files

0.4.34

2 files

0.4.33

2 files

0.4.32

2 files

0.4.31

2 files

0.4.30

2 files

0.4.29

2 files

0.4.28

2 files

0.4.27

2 files

0.4.26

2 files

0.4.25

2 files

0.4.24

2 files

0.4.23

2 files

0.4.22

2 files

0.4.21

2 files

0.4.20

2 files

0.4.19

2 files

0.4.18

2 files

0.4.17

2 files

0.4.16

2 files

This release

0.4.15 This release

2 files

0.4.14

2 files

0.4.13

2 files

0.4.12

2 files

0.4.11

2 files

0.4.10

2 files

0.4.9

2 files

0.4.8

2 files

0.4.7

2 files

0.4.5

2 files

0.4.4

2 files

0.4.3

2 files

0.4.2

2 files

0.4.1

2 files

0.4.0

2 files

0.3.46

2 files

0.3.45

2 files

0.3.44

2 files

0.3.43

2 files

0.3.42

2 files

0.3.41

2 files

0.3.40

2 files

0.3.39

2 files

0.3.38

2 files

0.3.37

2 files

0.3.36

2 files

0.3.35

2 files

0.3.34

2 files

0.3.33

2 files

0.3.32

2 files

0.3.31

2 files

0.3.30

2 files

0.3.29

2 files

0.3.28

2 files

0.3.27

2 files

0.3.26

2 files

0.3.25

2 files

0.3.24.1

2 files

0.3.24

2 files

0.3.23

2 files

0.3.22

2 files

0.3.21

2 files

0.3.20

2 files

0.3.19

2 files

0.3.18

2 files

0.3.17

2 files

0.3.16

2 files

0.3.15

2 files

0.3.14

2 files

0.3.13

2 files

0.3.12

2 files

0.3.11

2 files

0.3.10

2 files

0.3.9

2 files

0.3.8

2 files

0.3.7

2 files

Yanked

0.3.6

2 files

Yanked reason: badrelease

0.3.5

2 files

0.3.4

2 files

Yanked

0.3.4a1

2 files

0.3.3

2 files

0.3.2

2 files

Yanked

0.3.1

1 file

Pre-release

0.3.0a2

2 files

Pre-release

0.3.0a1

2 files

0.2.0

2 files

0.1.22

2 files

0.1.21

2 files

0.1.20

2 files

0.1.19

2 files

0.1.18

2 files

0.1.17

2 files

0.1.16

2 files

Pre-release

0.1.16.dev1

2 files

0.1.15

2 files

0.1.14

2 files

0.1.13

2 files

0.1.12

2 files

0.1.11

2 files

Pre-release

0.1.11.dev1

2 files

Pre-release

0.1.11.dev0

2 files

0.1.10

2 files

0.1.9

2 files

0.1.8

2 files

0.1.6

2 files

0.1.5

2 files

0.1.4

2 files

0.1.3

2 files

0.1.1

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page