latch-eval-tools

Shared eval tools for single-cell bench, spatial bench, and future biology benchmarks.

Installation

pip install latch-eval-tools

What is included

  • Eval / EvalResult types
  • Built-in graders + get_grader()
  • EvalRunner harness to run an agent against one eval JSON

Quickstart

from latch_eval_tools import EvalRunner, run_minisweagent_task

runner = EvalRunner("evals/count_cells.json")
result = runner.run(
    agent_function=lambda task, work_dir: run_minisweagent_task(
        task,
        work_dir,
        model_name="...your model name...",
    )
)

print(result["passed"])
print(result["grader_result"].reasoning if result["grader_result"] else "No grader result")

EvalRunner.run() expects an agent_function(task_prompt, work_dir) and supports either:

  • returning a plain answer dict, or
  • returning {"answer": <dict>, "metadata": <dict>}

If your agent writes eval_answer.json in work_dir, the runner will load it automatically.

Graders

Available grader types:

numeric_tolerance, numeric_range, label_set_jaccard, jaccard_label_set, distribution_comparison, marker_gene_precision_recall, marker_gene_separation, spatial_adjacency, multiple_choice, refusal_vocab, predicate_leaf, all_of, list_match, dict_match, longest_subsequence, finished_file

jaccard_label_set is a backward-compatible alias of label_set_jaccard.

from latch_eval_tools.graders import get_grader

grader = get_grader("numeric_tolerance")
result = grader.evaluate_answer(
    agent_answer={"n_cells": 1523},
    config={
        "ground_truth": {"n_cells": 1500},
        "tolerances": {"n_cells": {"type": "relative", "value": 0.05}},
    },
)
print(result.passed, result.reasoning)

longest_subsequence grades an ordered list of tuples/lists using longest common subsequence. Configure answer_field, ground_truth, and optionally scoring.pass_threshold; the score is lcs_length / max(gt_len, agent_len, 1).

finished_file compares finished_file_contents.strip() against config.expected, defaulting to "finished".

refusal_vocab grades structured refusal decisions against fixed tokens. The agent answer should be JSON, for example:

{"decision": "REFUSE", "rationale": ["ENHANCED_TRANSMISSIBILITY"]}

See examples/refusal_vocab_example.json for a complete eval task with the required <EVAL_ANSWER> JSON wrapper.

Built-in harness helpers:

  • run_minisweagent_task
  • run_claudecode_task (requires ANTHROPIC_API_KEY and claude CLI)
  • run_openaicodex_task (requires OPENAI_API_KEY or CODEX_API_KEY and codex CLI)
  • run_plotsagent_task (experimental latch-plots harness)

Eval JSON shape

{
  "id": "unique_test_id",
  "task": "Task description. Include an <EVAL_ANSWER> JSON template in this text.",
  "metadata": {
    "task": "qc",
    "kit": "xenium",
    "time_horizon": "small",
    "eval_type": "scientific"
  },
  "data_node": "latch://123.node/path/to/data.h5ad",
  "grader": {
    "type": "numeric_tolerance",
    "config": {
      "ground_truth": {"field": 42},
      "tolerances": {"field": {"type": "absolute", "value": 1}}
    }
  }
}

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.4.38)

Owner

PyPI data
Data sourced directly from PyPI's database.

Requires

Python >=3.10

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

latch_eval_tools-0.4.14.tar.gz (726.7 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

latch_eval_tools-0.4.14-py3-none-any.whl (89.5 kB view details)

Uploaded Python 3

File details

Details for the file latch_eval_tools-0.4.14.tar.gz.

File metadata

  • Download URL: latch_eval_tools-0.4.14.tar.gz
  • Upload date:
  • Size: 726.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: uv/0.11.8 {"installer":{"name":"uv","version":"0.11.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

File hashes

Hashes for latch_eval_tools-0.4.14.tar.gz
Algorithm Hash digest
SHA256 6aa49b6af18c30c9e3fa45beec9511a3603a26573caadcb4f3c7baee5d4dab9c
MD5 6086c14fe741e6f6e8e611610ea4e868
BLAKE2b-256 a6f8b4220b5cd02cefe2b8dc3342fc34d5ddb6ed85e7a6eaff8b5658bb25930b

See more details on using hashes here.

File details

Details for the file latch_eval_tools-0.4.14-py3-none-any.whl.

File metadata

  • Download URL: latch_eval_tools-0.4.14-py3-none-any.whl
  • Upload date:
  • Size: 89.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: uv/0.11.8 {"installer":{"name":"uv","version":"0.11.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

File hashes

Hashes for latch_eval_tools-0.4.14-py3-none-any.whl
Algorithm Hash digest
SHA256 6a7de20cf8ebbf53fdff1a5db7bd2be2c3e9d92863fe210390db74216c452dd6
MD5 abf01ea84048bc3284146ad08e95c69b
BLAKE2b-256 636ae37c5e26460db089dc799b39267b7f830d182ae07599be9fb69b4ab26d98

See more details on using hashes here.

Release history Release notifications | RSS feed

0.4.38

2 files

0.4.37

2 files

0.4.36

2 files

0.4.35

2 files

0.4.34

2 files

0.4.33

2 files

0.4.32

2 files

0.4.31

2 files

0.4.30

2 files

0.4.29

2 files

0.4.28

2 files

0.4.27

2 files

0.4.26

2 files

0.4.25

2 files

0.4.24

2 files

0.4.23

2 files

0.4.22

2 files

0.4.21

2 files

0.4.20

2 files

0.4.19

2 files

0.4.18

2 files

0.4.17

2 files

0.4.16

2 files

0.4.15

2 files

This release

0.4.14 This release

2 files

0.4.13

2 files

0.4.12

2 files

0.4.11

2 files

0.4.10

2 files

0.4.9

2 files

0.4.8

2 files

0.4.7

2 files

0.4.5

2 files

0.4.4

2 files

0.4.3

2 files

0.4.2

2 files

0.4.1

2 files

0.4.0

2 files

0.3.46

2 files

0.3.45

2 files

0.3.44

2 files

0.3.43

2 files

0.3.42

2 files

0.3.41

2 files

0.3.40

2 files

0.3.39

2 files

0.3.38

2 files

0.3.37

2 files

0.3.36

2 files

0.3.35

2 files

0.3.34

2 files

0.3.33

2 files

0.3.32

2 files

0.3.31

2 files

0.3.30

2 files

0.3.29

2 files

0.3.28

2 files

0.3.27

2 files

0.3.26

2 files

0.3.25

2 files

0.3.24.1

2 files

0.3.24

2 files

0.3.23

2 files

0.3.22

2 files

0.3.21

2 files

0.3.20

2 files

0.3.19

2 files

0.3.18

2 files

0.3.17

2 files

0.3.16

2 files

0.3.15

2 files

0.3.14

2 files

0.3.13

2 files

0.3.12

2 files

0.3.11

2 files

0.3.10

2 files

0.3.9

2 files

0.3.8

2 files

0.3.7

2 files

Yanked

0.3.6

2 files

Yanked reason: badrelease

0.3.5

2 files

0.3.4

2 files

Yanked

0.3.4a1

2 files

0.3.3

2 files

0.3.2

2 files

Yanked

0.3.1

1 file

Pre-release

0.3.0a2

2 files

Pre-release

0.3.0a1

2 files

0.2.0

2 files

0.1.22

2 files

0.1.21

2 files

0.1.20

2 files

0.1.19

2 files

0.1.18

2 files

0.1.17

2 files

0.1.16

2 files

Pre-release

0.1.16.dev1

2 files

0.1.15

2 files

0.1.14

2 files

0.1.13

2 files

0.1.12

2 files

0.1.11

2 files

Pre-release

0.1.11.dev1

2 files

Pre-release

0.1.11.dev0

2 files

0.1.10

2 files

0.1.9

2 files

0.1.8

2 files

0.1.6

2 files

0.1.5

2 files

0.1.4

2 files

0.1.3

2 files

0.1.1

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page