kt-audit

Before you trust an AUC, audit the evaluation.

kt-audit is a small, opinionated audit tool for knowledge-tracing (KT) experiments. It does not train models. It checks whether the evaluation setup behind a reported AUC is trustworthy.

Status: alpha (0.1.x). The scope is intentionally narrow; see docs/charter.md for the scope and non-goals.

Why

KT papers and libraries report AUC, but the evaluation protocol is often under-specified:

• Are train events strictly before validation/test events for each user?
• Does the same item appear in train and test?
• When an item has multiple knowledge components (KCs), are some KCs train-only and others test-only?
• Do the reported metrics match a recomputation from the released predictions?

kt-audit turns those questions into a machine-readable report you can attach to a paper artifact, a PR, or a model release.

Two 2026 measurements motivated the v0.2 checks. In a tutoring benchmark, a depth-4 tree over seven scalar features reproduced the gold diagnosis labels at 82.3 accuracy while the best LLM reached 42.0: the metric was measuring rule-following. In the same benchmark, a 3,000-token output budget silently truncated 14.5% of samples and cost 7 points of score.

These are not hypothetical. In pyKT, a widely used KT benchmark library, the authors themselves warn that "the training and test data has overlap and the experimental results cannot be compared" (pykt-team/pykt-toolkit#91), and data_process.py points valid and test at the same file.

Install

pip install kt-audit

Or with uv:

uv add kt-audit

From source:

git clone https://github.com/summeriver13/kt-audit
cd kt-audit
uv sync
uv run kt-audit --help

Requires Python 3.10+. Pin a specific version for reproducible paper artifacts.

Quickstart

Download the synthetic example and run the audit:

curl -fsSLO https://raw.githubusercontent.com/summeriver13/kt-audit/main/examples/leaky.csv
kt-audit run leaky.csv --out report

Or, from a source checkout:

uv run kt-audit run examples/leaky.csv --out report

Output:

verdict: fail
findings: 5 error, 0 warn
report: report/report.md

The command writes:

• report.json — stable, machine-readable.
• report.md — human-readable, with examples and limitations.

Exit codes: 0 pass/warn, 1 fail, 2 input/config error.

What it checks

Each finding has a severity, a count, and up to three examples.

cheap_baseline

KT claims that long history adds information. This check tests that claim directly: it builds eight features from past events only — prior count, prior accuracy, gap since the last event, same-item history, item popularity, KC history, and sequence position — trains a depth-4 decision tree on the training split, and compares its AUC with the reported predictions on the same rows.

• baseline AUC ≥ model AUC: error. The model adds nothing over cheap aggregates.
• model AUC − baseline AUC < baseline_warn_margin (0.02): warn.
• fewer than 50 labelled training rows, 30 evaluation rows, or one class: skipped.
• The comparison runs on the test split, or on valid when test is absent.

The numbers land in report.json under baselines and in a ## Baseline table.

degenerate

Thresholds are module constants, not config keys. Change them with a release.

Compare two runs

Two models that differ by 3 points on 200 rows may be indistinguishable. compare pairs the two prediction files on their interaction keys and bootstraps the AUC delta on the same resampled rows:

kt-audit compare baseline.csv candidate.csv --out comparison

verdict: b_better
delta: +0.4736 (95% CI [+0.3637, +0.5907])
report: comparison/comparison.md

• CI contains 0: tie. The sample cannot tell the two runs apart.
• Output: comparison.json and comparison.md.
• Defaults: 2000 rounds, seed 0, so the interval is reproducible.
• Inputs must share keys and agree on y_true; otherwise the command exits with code 2.

Case study

docs/case-study-assist09.md runs the tool on ASSISTments 2009 with a per-user chronological split. The v0.1 checks stay silent, which is correct; the cheap baseline reaches 0.7138, so a reported 0.771 passes and a reported 0.70 fails. Three severities changed as a result.

Provenance

report.json records tool_version and input.sha256, so an artifact stays readable after the input file is regenerated. The Markdown report prints both in its header.

Input

One row per interaction or prediction. CSV or JSONL.

See docs/data-contract.md for the full contract.

Python API

from kt_audit import AuditConfig, audit_path

report = audit_path("events.csv", AuditConfig(reported_auc=0.80))
print(report.verdict)
for finding in report.findings:
    print(finding.severity, finding.check, finding.count)

Design constraints

• Runtime dependencies: numpy, pandas, scikit-learn.
• CLI uses argparse; config uses JSON.
• Each check is a pure function over a prepared DataFrame.
• Synthetic examples only; no dataset is bundled.
• Findings are heuristics, not causal proof.

Development

uv sync --extra dev
uv run ruff check .
uv run ruff format --check .
uv run basedpyright
uv run pytest --cov=kt_audit --cov-report=term-missing
bash scripts/wheel-smoke.sh

Release

See docs/release.md and CHANGELOG.md. TestPyPI and PyPI publish through GitHub Actions trusted publishing.

Roadmap

• v0.2 (shipped): cheap_baseline, degenerate, compare, provenance fields. See docs/rfc-0.2.md.
• v0.3: pyKT adapter (covers the models wired in init_model.py, including the cold-start cskt), HTML report, offline judge-audit for saved LLM judgments, duplicate interactions, sequence truncation.

AI disclosure

AI tools are used in development. The maintainer reviews every check definition, test, and release. Findings generated by the tool are deterministic and do not call an LLM.

Contributing

See CONTRIBUTING.md. The v0.1 scope is frozen to the three checks above.

License

MIT. See LICENSE.