auditlint

Static analysis for paired and independently sampled statistical comparisons.

The checks operate on code shape rather than subject matter. A paired comparison is detected from two columns of the same object pulled out in parallel. An independently sampled comparison is detected from the same object filtered on the same column to two different literal values. The same logic applies to a clinical trial, a product experiment, a model version comparison, or a demographic comparison, and the test suite verifies all four. A plain checklist covering the same four checks without any tool is in CHECKLIST.md.

Checks

Test selection is checked against the apparent pairing structure of the arguments passed to it. An independent-samples test on matched observations discards their correlation and understates the precision of the comparison; a significant result is not undermined by this, a non-significant result is not well supported by it. A paired test on two independently filtered groups asserts a row correspondence the code has not established, since the arrays were built by filtering to different values rather than by aligning matched rows.

Fit and evaluation calls on the same object are checked for shared data. A distance, score, or threshold estimated from a sample and then applied to that same sample understates the variability the estimate would show on new data.

Hypothesis tests called more than once, including once per iteration of a loop, are checked for a false discovery rate or family-wise error correction. Reporting a family of tests without correction inflates the probability that at least one reaches significance by chance.

Scope

Analysis is single file and static. A data alignment step or leakage mitigation defined in a different module is not visible. A transform such as .copy() between a fit call and its evaluation produces different source text for what is, at runtime, the same data, and is not detected. A test call inside a helper function invoked from a loop is not recognised as looped, since loop membership is determined at the call site. These boundaries are encoded as assertions in the test suite.

Findings are JUST CANDIDATES for confirmation against the surrounding code.

Installation

pip install auditlint

Usage

auditlint path/to/script.py
auditlint path/to/directory
auditlint path/to/directory --fail-on blocking

--fail-on accepts blocking, flag, or none, and controls the exit status for use in a build pipeline. --json emits machine-readable output.

Claude Code plugin

/plugin marketplace add cindysteward/auditlint
/plugin install auditlint@auditlint
/auditlint:review path/to/directory

Installing the plugin also adds a file-save reviewer. Editing a Python file that calls a recognised statistical test, or that fits and evaluates a model, triggers the scanner automatically, gated on what the file does rather than on its name.

Extending recognised functions

A .auditlint.json file placed in a target file's directory, or any parent directory up to 6 levels up, adds to the recognised function names.

{
  "paired_tests": ["custom_paired_test"],
  "independent_tests": ["custom_rank_test"],
  "correction_markers": ["storey_qvalue"],
  "leakage_mitigation_markers": ["nested_cv"]
}

Pre-commit

- repo: https://github.com/cindysteward/auditlint
  rev: v0.1.0
  hooks:
    - id: auditlint

Security

The Claude Code hook runs a local Python script on every file save. It reads the file it was given, parses it with the standard library ast module, and writes a small cache under ~/.claude/auditlint_cache/. No network access, no modification of the file under review.

Licence

MIT.