inspect-evals-lint

Static checks for Inspect AI evaluations: file structure, test coverage conventions, best practices and sandbox image pinning. Documentation, including a page per rule, is at inspect-evals-lint.generality.org.

These checks began life as the autolint tool inside inspect_evals. They are packaged here so any repository of Inspect evaluations can run the same checks, including standalone repos built from the inspect-evals-template and submitted to the inspect_evals register.

Nothing is imported or executed from the evaluation being checked. Every check is static analysis over Python source (via ast), eval.yaml, compose files and pyproject.toml.

Install

uv add --dev inspect-evals-lint
# or
pip install inspect-evals-lint

Usage

inspect-evals-lint gpqa                     # one evaluation (or helper package, e.g. utils)
inspect-evals-lint gpqa utils               # several
inspect-evals-lint --all                    # every evaluation and helper package in the repo
inspect-evals-lint --all --select IEBP      # only the best-practice rules
inspect-evals-lint gpqa --ignore readme,IETS
inspect-evals-lint --all --output-format json > lint.json
inspect-evals-lint --all --output-format markdown   # a summary for a pull request comment
inspect-evals-lint --preset single-eval --task src/castle/castle.py   # the package holding a task file
inspect-evals-lint --list-rules             # every rule with its code, category, scope and summary
inspect-evals-lint --explain IEBP002        # one rule's documentation

Run it inside the project's environment (uv run inspect-evals-lint): external_dependencies maps import names to distributions through the packages installed there. --select replaces the configured selection for one run and --ignore adds to it; both take rule names, codes or code prefixes, comma-separated. With more than one package the text output ends with an overall summary, a per-rule compliance table and the failures grouped by rule.

--output-format json writes one document to stdout and sends progress to stderr, so the output can be piped straight into other tooling (docs/output.md has the schema). --output-format github writes one GitHub Actions annotation per finding instead, each message starting with the file, line and rule, so a lint step marks up the pull request and the job log still names every location; in an Actions job it also appends the Markdown summary to the job summary (GITHUB_STEP_SUMMARY). --output-format markdown writes, per package, a headline of rules met (with how many were met by suppression, when any were) and a per-category split, then a collapsible list of the rules not met, with warnings, or met by suppression, for a pull request comment or job summary. by_suppression(score) gives the headline's suffix for consumers building their own summaries.

--task <path> (repeatable) lints the package holding a task file instead of a named package: the evaluation is the directory holding the file, its parent is the source root, and enclosing packages form the import prefix, so a repository can be linted from a register entry's task_path without knowing its layout. A task file with no __init__.py beside it is a usage error naming the problem. Everything but the layout comes from the configuration or --preset. It carries schema_version, passed, run-wide summary counts and a packages list. Each package has its kind (eval or helper), rules (one entry per rule that ran, at its worst status, with indexes into that package's diagnostics), outcomes (one per rule that passed or did not apply) and diagnostics (one per finding, with rule, code, category (file_structure, code_quality, tests, best_practices or security), severity, status, message, file relative to the repository root, line, column and a hint). Every finding points at a file, and at a line where the finding is in a file's contents, so # inspect-evals-lint: ignore[<rule>] on that line suppresses it whatever the rule. JSON schema version 2 adds the security category and IESC rule prefix. Consumers of version 1 must update their category and code validation before upgrading. Existing rule codes and categories are unchanged. Schema version 3 counts suppressed rules as passing in score (docs/output.md has the details).

The repository root is the nearest pyproject.toml carrying a [tool.inspect-evals-lint] table (falling back to the nearest pyproject.toml, then the current directory). Pass --root to override.

Exit codes: 0 all checks passed (warnings, skips and suppressions count as passing), 1 at least one check failed, 2 usage or configuration error.

Configuration

Configuration lives in pyproject.toml. Pick a layout preset and override any field:

[tool.inspect-evals-lint]
preset = "multi-eval"   # or "single-eval" / "monorepo"

The presets describe how a repository is laid out. multi-eval is several evaluation packages side by side under src/, each registered by an entry point, as the inspect-evals-template produces. single-eval is one evaluation package that is the whole repository: tests directly under tests/ or in one tests/<name>/, the README at the repository root. monorepo is inspect_evals itself. When nothing names a preset (no [tool.inspect-evals-lint] table, or a table without preset) the source root decides: exactly one evaluation package gives single-eval, anything else multi-eval, and the run says which it chose and why. --preset overrides the table for one run.

The former names template and register still work for one release and print a notice naming their replacement. template was multi-eval. register was single-eval with eval-yaml-required = false, which is not a property of the layout but of the destination: the inspect_evals register entry holds the metadata. A repository that relies on that sets eval-yaml-required = false in its own table, and the register lint service passes it explicitly (REGISTER_CONFIG in the Python API). A fuller table:

[tool.inspect-evals-lint]
preset = "monorepo"
ignore = ["IETS004"]                                   # by name, code or prefix
exclude = ["src/inspect_evals/*/challenges/**"]         # sandbox code: never parsed
per-file-ignores = { "src/inspect_evals/*/data/**" = ["IEBP003"] }

[tool.inspect-evals-lint.allowlists.model_role_resolution]
moru = ["grader"]                                       # package = [keys the rule names]

[tool.inspect-evals-lint.allowlists.sandbox_image_pinning]
cybench = ["example/untagged"]

[tool.inspect-evals-lint.dockerfile_locking]                # a rule's own options
host-lock-coupling = "allow"

An allowlisted finding is reported as a warning with its message prefixed Allowlisted:, and an entry that no finding matches is itself a warning at pyproject.toml, so the list can only shrink. A table named after a rule holds that rule's options, merged over the preset's defaults key by key; each rule's page lists what it accepts. Removed keys (disabled-checks, non-eval-dirs, sandbox-image-allowlist, model-role-allowlist) are rejected with a message naming their replacement.

Only Python packages are linted: a sub-directory of source-root without an __init__.py (a README left behind after a move, a data directory) is skipped by --all and reported as a skip when named directly, so it needs no ignore-dirs entry. ignore-dirs is for packages you really do not want checked.

Helper packages

Shared code that evaluations import, such as inspect_evals' utils package, is not an evaluation but does most of the same things: it grades, it resolves models, it imports third-party packages. Directories listed in helper-dirs are linted with the checks that guard that behaviour (private_api_imports, score_constants, unscored_reason, get_model_location, model_role_resolution, sample_ids, task_overridable_defaults, sandbox_image_pinning, external_dependencies, tests_init and the custom_*_tests checks) and not with the ones about an evaluation's structure and registration (main_file, init_exports, readme, registry, eval_yaml, tests_exist, e2e_test, record_to_sample_test). Two checks adapt: external_dependencies requires a helper's module-level third-party imports to be in [project].dependencies, since every evaluation that imports the helper loads them, while imports inside a function, a try block or an if TYPE_CHECKING: block only need declaring in some group or isolated package; and the custom_*_tests checks look for a helper's tests anywhere under tests-root, not only in tests/<name>/. Allowlists are keyed by package name, so utils = ["grader"] under allowlists.model_role_resolution works for a helper too. A failing helper check fails the run like any other.

An upstream repository listed in the inspect_evals register is usually single-eval. Run it from outside the repo with inspect-evals-lint --root <clone> --preset single-eval --all --output-format json; the register lint service does the same through lint_task_files, which also leaves eval.yaml optional because the register entry holds the metadata.

For agents and tools

The documentation site is built to be read by machines as well as people:

• llms.txt is an index of the whole site in the llms.txt convention; llms-full.txt is every page concatenated.
• Every page is also served as raw Markdown at the same path with a .md suffix, for example rules/IEBP002.md.
• rules.json lists every rule with its code, name, category, scopes, summary and page URLs.
• Locally, inspect-evals-lint --explain <code> prints the same page text, --list-rules --output-format json lists the rules, and --output-format json gives findings with file, line, column and a hint (schema).

Suppressing a finding

Every finding has a file and, where it is in a file's contents, a line, so one comment syntax covers every rule:

• Line: # inspect-evals-lint: ignore[IEBP003] -- ids are row numbers on the offending line, or on any line of a multi-line statement (formatters move trailing comments inside parenthesised imports). Names, codes and code prefixes are accepted, comma-separated. In a Dockerfile, whose instructions take no trailing comment, put it on the line above the instruction.
• File: # inspect-evals-lint: ignore-file[IEBP003] -- <reason> within the first ten lines of the file.
• Paths: per-file-ignores in the configuration table, for whole directories.
• Not linted at all: exclude, for code that is shipped into a sandbox rather than run on the host.

Every comment gives a reason after --: why the finding is acceptable here, so the next reader can tell whether the decision still holds. A comment without one still suppresses the finding, and suppression_syntax (IECQ005) reports it as a warning.

The comment is namespaced with the tool's name rather than reusing ruff's # noqa: ruff reads every # noqa comment and warns about codes it does not know, so a shared spelling would turn each suppression into a ruff warning. Suppressed findings still appear in reports, marked [suppressed], and count as passing. A marker the linter does not read suppresses nothing and is reported as a warning by suppression_syntax (IECQ005) with the replacement in its hint: the former noautolint comments and .noautolint files, a bare ignore or ignore[], an ignore-file past the header, or a selector that names no rule.

Checks

The rule index lists every rule by category and links to a page per rule. The pages are in docs/rules/, generated from each rule's docstring by python -m inspect_evals_lint.docs; inspect-evals-lint --explain <code> prints the same text. Each rule is a decorated function in src/inspect_evals_lint/rules/: adding one means adding the function, its docstring, its references (the Inspect documentation sections that explain the convention, shown under See also on the rule's page and listed in rules.json; a rule with no upstream home for its convention has none) and its tests, and regenerating the docs (pre-commit checks they are current; the docs workflow also checks every reference resolves).

Python API

from pathlib import Path
from inspect_evals_lint import lint_repository, load_config

root = Path(".")
run = lint_repository(root, load_config(root))      # RunReport: every evaluation and helper package
for package in run.packages:
    print(package.name, package.kind, package.passed(), package.summary())
    for d in package.diagnostics:
        print(f"  {d.rule.code} {d.location}: {d.message}")

lint_package(root, name, config) lints one package and returns a PackageReport. lint_task_files(root, ["src/castle/castle.py"]) lints the packages holding task files (one per distinct layout, see task_layout) and raises UnsupportedLayoutError for a bare module. report.score() and run.score() give a Score: every rule counted once at the worst status it reported (fail, warn, suppressed, pass, skip in that order), with passing (pass, warn and suppressed), applicable (everything but skip) and a per-category split, the numbers the register badges show. render_markdown(run, source_link=...) produces the Markdown summary, with locations linked wherever source_link(path, line) returns a URL. rules() lists every registered Rule; get_rule("IEBP002") or get_rule("model_role_resolution") looks one up. run.to_dict() is the JSON document described in docs/output.md.

Development

uv sync
uv run pre-commit install   # optional: run the lint stack on every commit
uv run pytest                 # per-rule tests live under tests/rules/
uv run basedpyright src

Each pull request adds a changelog fragment rather than editing CHANGELOG.md, so concurrent PRs don't conflict. Run uv run scriv create, uncomment the sections that apply in the new file under changelog.d/, and commit it with the change, or delete it if the change needs no entry. To release, run the Prepare release workflow from the Actions tab. It bumps version in pyproject.toml, collects the fragments into CHANGELOG.md, and opens a release pull request, whose CI starts once you click Approve workflows to run on it; merging it tags the release. It needs Settings → Actions → General → Allow GitHub Actions to create and approve pull requests. By hand, the same is uv version --bump minor and uv run scriv collect.

docs/rules/ and the configuration table above are generated by uv run python -m inspect_evals_lint.docs; pre-commit fails if they are out of date. The pages that list every rule (the site's front page, the rule index, llms.txt, llms-full.txt and rules.json) are built with the site by mkdocs_hooks.py and not committed, so pull requests that add rules don't conflict on them. uv run --group docs mkdocs serve previews the site, which docs.yml publishes from main. Linting (ruff, zizmor, mdformat) runs via pre-commit; CI runs the same stack plus basedpyright and pytest via the shared python-ci reusable workflow.

Releasing

See RELEASING.md.