This release has been yanked by its maintainers, and will be ignored by installers, except when explicitly specified.
Reason given by maintainers: Withdrawn; not for use
rankcheck
rankcheck counts how many real ranks a leaderboard has and how many independent signals a panel of models gives.
It is for people who read, publish or build leaderboards and model ensembles, and who want to know which "we beat X" results would hold up under a paired test.
It accompanies the paper "How Many Independent Signals Do AI Leaderboards and AI Forecasters Carry?" (Apart AI x Epistemics sprint, November 2026).
pip install rankcheck==0.1.0
$ rankcheck tiers --demo swebench-verified
board: SWE-bench Verified, population results (SWE-bench/experiments@40f164d5 ...)
systems: 135 items: 500 test: exact McNemar correction: Holm (FWER 0.05)
adjacent ranks separable: 0 of 134 (Holm); 0 of 134 (Benjamini-Hochberg, q 0.05)
separable tiers: 12 for 135 systems (walk from the top, Bonferroni alpha 3.73e-04 per test)
tier 1: 17 systems, head score 0.792: ...
records (a new best by date) separable from the previous record: 3 of 32 (Holm)
Status: 0.1.0, Python 3.10 or newer, depends on numpy and rfc8785. verify-seal also needs the openssl command.
Commands
| Command | What it answers |
|---|---|
rankcheck tiers board.csv |
For a board with columns model,item,score: are neighbouring ranks distinguishable by a paired test over items (exact McNemar for 0/1 items, paired sign-flip otherwise, Holm correction), and how many separable tiers does the board hold? --pairs prints every pair, --json everything. |
rankcheck tiers --demo swebench-verified |
The same on the bundled per-instance SWE-bench Verified results (--population results, all or bash-only). |
rankcheck ranks board.csv |
Simultaneous 95% confidence sets for every system's rank (Mogstad, Romano, Shaikh and Wilhelm 2024, stepwise, item bootstrap; --bootstrap gaussian for the csranks parametric bootstrap). On 0/1 boards also the exact construction of Neuhof and Benjamini (Holm over all one-sided exact paired tests). --check-examples recomputes the paper's Tables 1 and 2 and compares. |
rankcheck indep forecasts.csv |
For columns model,item,p (optional baseline, cluster, outcome; re-asks as model#r2): the participation ratio and exchangeable effective number of forecasters with item-bootstrap intervals, and, when a reference forecast is given, the calibrated number of forecasters whose departures from the reference are independent. --demo runs a synthetic panel with a known answer. |
rankcheck verify-seal <bundle> |
For a sealed forecasting trial: rebuilds each wave's Merkle root, checks every RFC 3161 token against pinned authority roots (quorum of 2), and checks the wave's entry in the public append-only registry and its Rekor record. Prints PASS, FAIL or NOT RUN per check. |
What the tests mean
Each system on a public per-item board is usually one run. The paired test covers item sampling only: if two systems could not be told apart on these items, a rerun would not make them easier to tell apart.
Data
The bundled SWE-bench Verified board is built from the public per-instance results in
SWE-bench/experiments at commit 40f164d5: an instance counts as
resolved when it is in the official resolved list.
The bundled PISA 2018 reading table (OECD countries, used by --check-examples) is the pisa dataset of the
csranks R package 1.3.0, from OECD PISA 2018.
Prior work
rankcheck was written during Apart Research's AI x Epistemics research sprint (13 to 15 November 2026). None of its
own code is preexisting. The files under src/rankcheck/_sealed/ are byte-identical copies of estimator files sealed
during the sprint; their SHA-256 digests are in src/rankcheck/_sealed/SEALED.json.
Preexisting third-party code it depends on: numpy and rfc8785; optionally rfc3161-client, cryptography, scipy and
pytest. Bundled third-party data: SWE-bench/experiments at commit 40f164d5 and the csranks PISA 2018 table.
License
Metadata
Release files for rankcheck 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| rankcheck-0.1.0.tar.gz | 91.9 kB | Without provenance | Details for rankcheck-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| rankcheck-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for rankcheck-0.1.0-py3-none-any.whl |
Total release size: 183.6 kB
Release files / rankcheck-0.1.0.tar.gz
| Download URL | rankcheck-0.1.0.tar.gz |
|---|---|
| Size | 91.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
10f18019a216d5fa68b0a81797fe63a90545ddad30295473bce3d4afa9833ea2
|
|
BLAKE2b-256 checksum How to use checksums |
f46445530b617847e83a5f9153b31e4ed9c23262cd484233ccfbe90526d778e8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / rankcheck-0.1.0-py3-none-any.whl
| Download URL | rankcheck-0.1.0-py3-none-any.whl |
|---|---|
| Size | 91.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
dcb0de08dc8182423c5420089111ed63f9eefe19c33f30ad6787a484fe0207a8
|
|
BLAKE2b-256 checksum How to use checksums |
fdee848b7955b387058c16a1ea39894fd8ef9db18a74c4e2991fb31805ea9887
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for rankcheck 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.