inspect-evals-lint

Static checks for Inspect AI evaluations: file structure, test coverage conventions, best practices and sandbox image pinning.

These checks began life as the autolint tool inside inspect_evals. They are packaged here so any repository of Inspect evaluations can run the same checks, including standalone repos built from the inspect-evals-template and submitted to the inspect_evals register.

Nothing is imported or executed from the evaluation being checked. Every check is static analysis over Python source (via ast), eval.yaml, compose files and pyproject.toml.

Install

uv add --dev inspect-evals-lint
# or
pip install inspect-evals-lint

Usage

inspect-evals-lint <eval_name>            # one evaluation
inspect-evals-lint --all-evals            # every evaluation in the repo
inspect-evals-lint --all-evals --summary-only
inspect-evals-lint --check-summary        # per-check compliance across evals
inspect-evals-lint <eval_name> --check registry
inspect-evals-lint --all-evals --json > lint.json
inspect-evals-lint --list-checks

--json writes one document to stdout and sends progress to stderr, so the output can be piped straight into other tooling. It carries passed, run-wide summary counts and, per evaluation, every check's status, category (the CHECKS.md section: file_structure, code_quality, tests or best_practices), message, file (relative to the repository root when possible) and line.

The repository root is the nearest pyproject.toml carrying a [tool.inspect-evals-lint] table (falling back to the nearest pyproject.toml, then the current directory). Pass --root to override.

Exit codes: 0 all checks passed (warnings, skips and suppressions count as passing), 1 at least one check failed, 2 usage or configuration error.

Configuration

Configuration lives in pyproject.toml. Pick a layout preset and override any field:

[tool.inspect-evals-lint]
preset = "template"   # or "monorepo" / "register"

Without a [tool.inspect-evals-lint] table the template preset is used. --preset overrides the table for one run.

The register preset is for an upstream repository listed in the inspect_evals register: one evaluation, tests directly under tests/, the README at the repository root, and metadata held by the register entry rather than an eval.yaml in the repo. Run it from outside the repo with inspect-evals-lint --root <clone> --preset register --all-evals --json.

Suppressing a check

• Line: # noautolint: <check_name> on the offending line (checks that report per-site results: private_api_imports, get_model_location).
• File: # noautolint-file: <check_name> within the first ten lines of a file.
• Directory: a .noautolint file in a sub-directory listing check names, one per line. Files under that directory are also excluded from AST-based checks.
• Evaluation: a .noautolint file in the evaluation directory listing check names.

Checks

See docs/CHECKS.md for the full list with the reasoning behind each check.

Python API

from pathlib import Path
from inspect_evals_lint import lint_evaluation, load_config, get_all_eval_names

root = Path(".")
config = load_config(root)
for name in get_all_eval_names(root, config):
    report = lint_evaluation(root, name, config)
    print(name, report.passed(), report.summary())

Development

uv sync
uv run pre-commit install   # optional: run the lint stack on every commit
uv run pytest
uv run basedpyright src

Linting (ruff, zizmor, mdformat) runs via pre-commit; CI runs the same stack plus basedpyright and pytest via the shared python-ci reusable workflow.

Releasing

See RELEASING.md.