kt-audit

Before you trust an AUC, audit the evaluation.

kt-audit is a small, opinionated audit tool for knowledge-tracing (KT) experiments. It does not train models. It checks whether the evaluation setup behind a reported AUC is trustworthy.

Status: alpha (0.1.0). The scope is intentionally narrow; see docs/charter.md for the scope and non-goals.

Why

KT papers and libraries report AUC, but the evaluation protocol is often under-specified:

• Are train events strictly before validation/test events for each user?
• Does the same item appear in train and test?
• When an item has multiple knowledge components (KCs), are some KCs train-only and others test-only?
• Do the reported metrics match a recomputation from the released predictions?

kt-audit turns those questions into a machine-readable report you can attach to a paper artifact, a PR, or a model release.

Install

Not on PyPI yet. From source:

git clone https://github.com/summeriver13/kt-audit
cd kt-audit
uv sync
uv run kt-audit --help

Or with pip:

pip install -e .

Requires Python 3.10+.

Quickstart

uv run kt-audit run examples/leaky.csv --out /tmp/kt-audit-report

Output:

verdict: fail
findings: 6 error, 2 warn
report: /tmp/kt-audit-report/report.md

The command writes:

• report.json — stable, machine-readable.
• report.md — human-readable, with examples and limitations.

Exit codes: 0 pass/warn, 1 fail, 2 input/config error.

What it checks

Each finding has a severity, a count, and up to three examples.

Input

One row per interaction or prediction. CSV or JSONL.

See docs/data-contract.md for the full contract.

Python API

from kt_audit import AuditConfig, audit_path

report = audit_path("events.csv", AuditConfig(reported_auc=0.80))
print(report.verdict)
for finding in report.findings:
    print(finding.severity, finding.check, finding.count)

Design constraints

• Runtime dependencies: numpy, pandas, scikit-learn.
• CLI uses argparse; config uses JSON.
• Each check is a pure function over a prepared DataFrame.
• Synthetic examples only; no dataset is bundled.
• Findings are heuristics, not causal proof.

Development

uv sync --extra dev
uv run ruff check .
uv run ruff format --check .
uv run basedpyright
uv run pytest --cov=kt_audit --cov-report=term-missing
bash scripts/wheel-smoke.sh

Release

See docs/release.md. TestPyPI and PyPI publish through GitHub Actions trusted publishing.

Roadmap

• v0.2: pyKT adapter, bootstrap AUC intervals, HTML report.
• v0.3: duplicate interactions, cold-start, sequence truncation, label leakage.

AI disclosure

AI tools are used in development. The maintainer reviews every check definition, test, and release. Findings generated by the tool are deterministic and do not call an LLM.

Contributing

See CONTRIBUTING.md. The v0.1 scope is frozen to the three checks above.

License

MIT. See LICENSE.