inspect-evals-lint

Static checks for Inspect AI evaluations: file structure, test coverage conventions, best practices and sandbox image pinning.

These checks began life as the autolint tool inside inspect_evals. They are packaged here so any repository of Inspect evaluations can run the same checks, including standalone repos built from the inspect-evals-template and submitted to the inspect_evals register.

Nothing is imported or executed from the evaluation being checked. Every check is static analysis over Python source (via ast), eval.yaml, compose files and pyproject.toml.

Install

uv add --dev inspect-evals-lint
# or
pip install inspect-evals-lint

Usage

inspect-evals-lint <eval_name>            # one evaluation (or helper package, e.g. utils)
inspect-evals-lint --all-evals            # every evaluation and helper package in the repo
inspect-evals-lint --all-evals --summary-only
inspect-evals-lint --check-summary        # per-check compliance across evals
inspect-evals-lint <eval_name> --check registry
inspect-evals-lint --all-evals --json > lint.json
inspect-evals-lint --list-checks

--json writes one document to stdout and sends progress to stderr, so the output can be piped straight into other tooling. It carries passed, run-wide summary counts and, per package, its kind (eval or helper) and every check's status, category (the CHECKS.md section: file_structure, code_quality, tests or best_practices), message, file (relative to the repository root when possible) and line. Evaluations are listed under evaluations (counted by evaluations_total / evaluations_passed) and helper packages under helpers (helpers_total / helpers_passed), so a consumer that only knows evaluations keeps reading the same list. The set of categories is stable: a new check always joins one of the four, because badge and dashboard tooling keys on them.

The repository root is the nearest pyproject.toml carrying a [tool.inspect-evals-lint] table (falling back to the nearest pyproject.toml, then the current directory). Pass --root to override.

Exit codes: 0 all checks passed (warnings, skips and suppressions count as passing), 1 at least one check failed, 2 usage or configuration error.

Configuration

Configuration lives in pyproject.toml. Pick a layout preset and override any field:

[tool.inspect-evals-lint]
preset = "template"   # or "monorepo" / "register"

Without a [tool.inspect-evals-lint] table the template preset is used. --preset overrides the table for one run.

Only Python packages are linted: a sub-directory of source-root without an __init__.py (a README left behind after a move, a data directory) is skipped by --all-evals and reported as a skip when named directly, so it needs no ignore-dirs entry. ignore-dirs is for packages you really do not want checked.

Helper packages

Shared code that evaluations import, such as inspect_evals' utils package, is not an evaluation but does most of the same things: it grades, it resolves models, it imports third-party packages. Directories listed in helper-dirs are linted with the checks that guard that behaviour (private_api_imports, score_constants, unscored_reason, get_model_location, model_role_resolution, sample_ids, task_overridable_defaults, sandbox_image_pinning, external_dependencies, tests_init and the custom_*_tests checks) and not with the ones about an evaluation's structure and registration (main_file, init_exports, readme, registry, eval_yaml, tests_exist, e2e_test, record_to_sample_test). Two checks adapt: external_dependencies requires a helper's module-level third-party imports to be in [project].dependencies, since every evaluation that imports the helper loads them, while imports inside a function, a try block or an if TYPE_CHECKING: block only need declaring in some group or isolated package; and the custom_*_tests checks look for a helper's tests anywhere under tests-root, not only in tests/<name>/. Allowlists are keyed by directory name, so utils = ["grader"] under model-role-allowlist works for a helper too. A failing helper check fails the run like any other.

The register preset is for an upstream repository listed in the inspect_evals register: one evaluation, tests directly under tests/, the README at the repository root, and metadata held by the register entry rather than an eval.yaml in the repo. Run it from outside the repo with inspect-evals-lint --root <clone> --preset register --all-evals --json.

Suppressing a check

• Line: # noautolint: <check_name> on the offending line (checks that report per-site results: private_api_imports, get_model_location).
• File: # noautolint-file: <check_name> within the first ten lines of a file.
• Directory: a .noautolint file in a sub-directory listing check names, one per line. Files under that directory are also excluded from AST-based checks.
• Evaluation: a .noautolint file in the evaluation directory listing check names.

Checks

See docs/CHECKS.md for the full list with the reasoning behind each check.

Python API

from pathlib import Path
from inspect_evals_lint import (
    get_all_eval_names,
    get_all_helper_names,
    lint_evaluation,
    load_config,
)

root = Path(".")
config = load_config(root)
for name in (*get_all_eval_names(root, config), *get_all_helper_names(root, config)):
    report = lint_evaluation(root, name, config)
    print(name, report.kind, report.passed(), report.summary())

Development

uv sync
uv run pre-commit install   # optional: run the lint stack on every commit
uv run pytest
uv run basedpyright src

Linting (ruff, zizmor, mdformat) runs via pre-commit; CI runs the same stack plus basedpyright and pytest via the shared python-ci reusable workflow.

Releasing

See RELEASING.md.