inspect-test-utils 1.4.0

Simple models and tasks for integration testing

inspect-test-utils

A small collection of tasks, scorers, and a simple model for use with the Inspect AI framework. It is designed to support integration/acceptance tests, demos, and reproductions by providing:

• Ready-made Tasks that exercise common evaluation patterns (simple generation, numeric closeness, failure injection, and sandbox configuration).
• Scorers for deterministic or parameterized scoring (including hardcoded outputs and a logarithmic closeness score).
• A hardcoded ModelAPI implementation that can deterministically emit tool calls and/or final answers, useful for testing tool-calling flows without hitting external APIs.

Passing arguments Most tasks and the hardcoded model accept parameters. With the Inspect CLI you can pass them via --task-arg and --model-arg repeatedly:

• Example: make the task generate 3 samples and set a numeric target for guessing: inspect eval inspect_test_utils/guess_number
  --task-arg sample_count=3
  --task-arg target=42.7
  --model hardcoded --model-arg answer=42.6

What’s included

• Tasks (inspect_test_utils.tasks)

  • say_hello(sample_count=1): Simple task; expects a response that includes "hello".
  • guess_number(sample_count=1, target="42.7"): Uses a logarithmic closeness scorer for numeric answers.
  • hardcoded_score(sample_count=10, hardcoded_score=None, hardcoded_score_by_sample_id_and_epoch=None): Scores are injected from parameters; useful for testing aggregations and edge cases (including NaN).
  • sometimes_fails_setup(sample_count=10, fail_setup_on_epochs=None, failure_rate=0.2): Randomly raises during setup via a failing solver; useful to test retry/resume behavior.
  • sometimes_fails_scoring(sample_count=10, fail_score_on_epochs=None, failure_rate=0.2): Randomly raises during scoring; useful to test scorer error handling.
  • configurable_sandbox(sample_count=1, cpu=0.5, memory="2G", storage="2G", gpu=None, gpu_model=None, allow_internet=False): A task with runtime configurable sandbox.

• Scorers (inspect_test_utils.scorers)

  • failing_scorer(fail_on_epochs=None, failure_rate=0.2): Raises errors at a controlled rate for selected epochs.
  • closeness_log(): Scores 1.0 for exact equality, otherwise 1/(1+log1p(relative_error)) for numeric strings.
  • hardcoded_scorer(hardcoded_score=None, hardcoded_score_by_sample_id_and_epoch=None): Returns pre-specified Score objects or looks them up by sample id and epoch.

• Model (inspect_test_utils.hardcoded)

  • hardcoded: A ModelAPI that can emit a sequence of tool calls (e.g., bash) for a number of repetitions and then submit a final answer. Parameters include:
    • answer: final answer string (default: "done").
    • repetitions: how many tool-call "turns" before submitting.
    • tool_calls: list of tool calls or shell strings (e.g., ["echo hi", "ls -la"]) to simulate; defaults to none.
    • delay: optional delay (seconds) before returning each model output.

Testing checkpoint/resume of an (agent, task) pair

inspect_test_utils includes a crash/resume harness that verifies an (agent, task) pair correctly checkpoints and resumes after a mid-run or scoring crash. Use it with any task that calls react() (or another checkpointer-aware solver) and has a checkpoint trigger configured.

from inspect_ai import Task
from inspect_ai.agent import react
from inspect_ai.dataset import Sample
from inspect_ai.scorer import includes
from inspect_ai.util import CheckpointSampleConfig

from inspect_test_utils import (
    run_resume_test,
    after_turns,
    at_scoring,
    assert_resumed,
    assert_agent_not_restarted,
    assert_score_recovered,
)

task = Task(
    dataset=[
        Sample(
            id="s1",
            input="go",
            target="done",
            checkpoint=CheckpointSampleConfig(sandbox_paths={"default": ["/root"]}),
        )
    ],
    solver=react(...),
    scorer=includes(),
    sandbox="docker",
)

# Crash mid-run (after the 2nd sandbox exec) and assert the agent resumed:
r = run_resume_test(task, crash=after_turns(2), compute_baseline=False)
assert_resumed(r)

# Crash at the first scoring call and assert the agent was NOT re-run (scoring-only resume):
task_no_sandbox = Task(
    dataset=[Sample(id="s1", input="hi", target="hi")],
    solver=react(...),
    scorer=includes(),
)
r = run_resume_test(task_no_sandbox, crash=at_scoring())
assert_resumed(r)
assert_agent_not_restarted(r)  # agent loop skipped on scoring resume
assert_score_recovered(r)      # score matches baseline

Both after_turns(n) (crash after the n-th sandbox exec) and at_scoring() (crash at the first scorer call) are supported. after_turns requires compute_baseline=False because the exec patch is incompatible with a second in-process checkpointed eval.

run_resume_test uses an in-process soft crash (CrashInjected exception + eval_set retry). For a true os._exit crash as in a real k8s/hawk deployment, compose crash_after_exec(n, hard=True) into your task's solver chain — this is the resume_probe pattern generalised to any agent via the shared exec seam. Because hard=True calls os._exit, it cannot run inside pytest (it would kill the harness); it is intended for real eval-set jobs where the platform handles restart and resume.

Installation

pip install inspect-test-utils

License

MIT. See LICENSE.