latch-eval-tools

Shared eval tools for single-cell bench, spatial bench, and future biology benchmarks.

Installation

pip install latch-eval-tools

What is included

  • Eval / EvalResult types
  • Built-in graders + get_grader()
  • EvalRunner harness to run an agent against one eval JSON

Quickstart

from latch_eval_tools import EvalRunner, run_minisweagent_task

runner = EvalRunner("evals/count_cells.json")
result = runner.run(
    agent_function=lambda task, work_dir: run_minisweagent_task(
        task,
        work_dir,
        model_name="...your model name...",
    )
)

print(result["passed"])
print(result["grader_result"].reasoning if result["grader_result"] else "No grader result")

EvalRunner.run() expects an agent_function(task_prompt, work_dir) and supports either:

  • returning a plain answer dict, or
  • returning {"answer": <dict>, "metadata": <dict>}

If your agent writes eval_answer.json in work_dir, the runner will load it automatically.

Graders

Available grader types:

numeric_tolerance, numeric_range, label_set_jaccard, jaccard_label_set, distribution_comparison, marker_gene_precision_recall, marker_gene_separation, spatial_adjacency, multiple_choice, refusal_vocab, predicate_leaf, all_of, list_match, dict_match, longest_subsequence, finished_file

jaccard_label_set is a backward-compatible alias of label_set_jaccard.

from latch_eval_tools.graders import get_grader

grader = get_grader("numeric_tolerance")
result = grader.evaluate_answer(
    agent_answer={"n_cells": 1523},
    config={
        "ground_truth": {"n_cells": 1500},
        "tolerances": {"n_cells": {"type": "relative", "value": 0.05}},
    },
)
print(result.passed, result.reasoning)

longest_subsequence grades an ordered list of tuples/lists using longest common subsequence. Configure answer_field, ground_truth, and optionally scoring.pass_threshold; the score is lcs_length / max(gt_len, agent_len, 1).

finished_file compares finished_file_contents.strip() against config.expected, defaulting to "finished".

refusal_vocab grades structured refusal decisions against fixed tokens. The agent answer should be JSON, for example:

{"decision": "REFUSE", "rationale": ["ENHANCED_TRANSMISSIBILITY"]}

See examples/refusal_vocab_example.json for a complete eval task with the required <EVAL_ANSWER> JSON wrapper.

Built-in harness helpers:

  • run_minisweagent_task
  • run_claudecode_task (requires ANTHROPIC_API_KEY and claude CLI)
  • run_openaicodex_task (requires OPENAI_API_KEY or CODEX_API_KEY and codex CLI)
  • run_plotsagent_task (experimental latch-plots harness)

Eval JSON shape

{
  "id": "unique_test_id",
  "task": "Task description. Include an <EVAL_ANSWER> JSON template in this text.",
  "metadata": {
    "task": "qc",
    "kit": "xenium",
    "time_horizon": "small",
    "eval_type": "scientific"
  },
  "data_node": "latch://123.node/path/to/data.h5ad",
  "grader": {
    "type": "numeric_tolerance",
    "config": {
      "ground_truth": {"field": 42},
      "tolerances": {"field": {"type": "absolute", "value": 1}}
    }
  }
}

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.4.38)

Owner

PyPI data
Data sourced directly from PyPI's database.

Requires

Python >=3.10

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

latch_eval_tools-0.4.17.tar.gz (738.7 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

latch_eval_tools-0.4.17-py3-none-any.whl (96.6 kB view details)

Uploaded Python 3

File details

Details for the file latch_eval_tools-0.4.17.tar.gz.

File metadata

  • Download URL: latch_eval_tools-0.4.17.tar.gz
  • Upload date:
  • Size: 738.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

File hashes

Hashes for latch_eval_tools-0.4.17.tar.gz
Algorithm Hash digest
SHA256 1a9b1e324f852edc1ff829d061c2679e8c513d09002758d1683644e1ec897a96
MD5 ac536760a62c02d151430f46bac25519
BLAKE2b-256 8e6c23763a5a405aaaea68619c89a940c2cf8f65502aed77a277073a7ca29929

See more details on using hashes here.

File details

Details for the file latch_eval_tools-0.4.17-py3-none-any.whl.

File metadata

  • Download URL: latch_eval_tools-0.4.17-py3-none-any.whl
  • Upload date:
  • Size: 96.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: uv/0.11.19 {"installer":{"name":"uv","version":"0.11.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

File hashes

Hashes for latch_eval_tools-0.4.17-py3-none-any.whl
Algorithm Hash digest
SHA256 44427b5ea85ac30f7ccb346bb3c7440c6f6d1e27c68d64463100011995cf1c94
MD5 27b5eb84666f92494b72f31996e2b3cb
BLAKE2b-256 99815d197fe7a55ade09d7416577c06a10daf61721953bad103d8c6aebbe8954

See more details on using hashes here.

Release history Release notifications | RSS feed

0.4.38

2 files

0.4.37

2 files

0.4.36

2 files

0.4.35

2 files

0.4.34

2 files

0.4.33

2 files

0.4.32

2 files

0.4.31

2 files

0.4.30

2 files

0.4.29

2 files

0.4.28

2 files

0.4.27

2 files

0.4.26

2 files

0.4.25

2 files

0.4.24

2 files

0.4.23

2 files

0.4.22

2 files

0.4.21

2 files

0.4.20

2 files

0.4.19

2 files

0.4.18

2 files

This release

0.4.17 This release

2 files

0.4.16

2 files

0.4.15

2 files

0.4.14

2 files

0.4.13

2 files

0.4.12

2 files

0.4.11

2 files

0.4.10

2 files

0.4.9

2 files

0.4.8

2 files

0.4.7

2 files

0.4.5

2 files

0.4.4

2 files

0.4.3

2 files

0.4.2

2 files

0.4.1

2 files

0.4.0

2 files

0.3.46

2 files

0.3.45

2 files

0.3.44

2 files

0.3.43

2 files

0.3.42

2 files

0.3.41

2 files

0.3.40

2 files

0.3.39

2 files

0.3.38

2 files

0.3.37

2 files

0.3.36

2 files

0.3.35

2 files

0.3.34

2 files

0.3.33

2 files

0.3.32

2 files

0.3.31

2 files

0.3.30

2 files

0.3.29

2 files

0.3.28

2 files

0.3.27

2 files

0.3.26

2 files

0.3.25

2 files

0.3.24.1

2 files

0.3.24

2 files

0.3.23

2 files

0.3.22

2 files

0.3.21

2 files

0.3.20

2 files

0.3.19

2 files

0.3.18

2 files

0.3.17

2 files

0.3.16

2 files

0.3.15

2 files

0.3.14

2 files

0.3.13

2 files

0.3.12

2 files

0.3.11

2 files

0.3.10

2 files

0.3.9

2 files

0.3.8

2 files

0.3.7

2 files

Yanked

0.3.6

2 files

Yanked reason: badrelease

0.3.5

2 files

0.3.4

2 files

Yanked

0.3.4a1

2 files

0.3.3

2 files

0.3.2

2 files

Yanked

0.3.1

1 file

Pre-release

0.3.0a2

2 files

Pre-release

0.3.0a1

2 files

0.2.0

2 files

0.1.22

2 files

0.1.21

2 files

0.1.20

2 files

0.1.19

2 files

0.1.18

2 files

0.1.17

2 files

0.1.16

2 files

Pre-release

0.1.16.dev1

2 files

0.1.15

2 files

0.1.14

2 files

0.1.13

2 files

0.1.12

2 files

0.1.11

2 files

Pre-release

0.1.11.dev1

2 files

Pre-release

0.1.11.dev0

2 files

0.1.10

2 files

0.1.9

2 files

0.1.8

2 files

0.1.6

2 files

0.1.5

2 files

0.1.4

2 files

0.1.3

2 files

0.1.1

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page