Skip to main content

Certified Corruption Budget

Attach a certified corruption budget to leaderboard claims.

Every ranking claim gets a number B_hat: with probability at least 1 - alpha, at every time t, either the claim holds, or more than B_hat of the first t records were corrupted. The guarantee holds even if an attacker watches every published certificate and corrupts votes in bursts.

This is a product implementation of:

Hamed Khosravi and Xiaoming Huo, Certified Corruption Budgets: Anytime-Valid Leaderboard Claims under Adaptive Rigging, arXiv:2610.10597, 2026. https://arxiv.org/abs/2610.10597

This is an independent implementation. It is not affiliated with, endorsed by, or reviewed by the paper's authors.

Status

Local-first Python library and CLI. Edge and lead certificates, insertion and replacement costs, batch and streaming, selection correction, closed-testing and Bonferroni leaderboards, attack replays, a human table, a stable ccb/v1 JSON envelope, and a self-contained HTML report are implemented and tested. Registry publication is pending the owner's decisions.

Install

uv build
pip install dist/certified_corruption_budget-0.1.0-py3-none-any.whl

Python 3.11 or newer; NumPy is the only runtime dependency.

Example

$ ccb certify --input records.jsonl --pair model-x,model-y
pair          model-x > model-y
claim         edge
cost          replacement
alpha         0.05
B_rep         2
certified_at  13

At alpha=0.05, the edge claim that model-x has had an advantage over model-y holds unless more than 2 of the first 20 records were forged or altered (replacement cost).

What it does

  • Ingests pairwise records from JSONL or CSV.
  • Emits edge and lead certificates with insertion or replacement costs.
  • Emits simultaneous certificates for a whole leaderboard.
  • Corrects for selecting the best of V private model variants.
  • Replays burst, flip, and spread attacks against the certificate and the paper's baselines.
  • Produces a human table, a stable JSON envelope, and a static HTML report.

CLI

ccb certify     --input examples/records.jsonl --pair a,b --claim edge --cost replacement --alpha 0.05
ccb certify     --input examples/records.jsonl --pair a,b --json
ccb leaderboard --input examples/records.jsonl --method closed --cost replacement --alpha 0.05
ccb replay      --input examples/null.jsonl --attack flip --budget 100 --replays 1000 --json
ccb demo        --input examples/records.jsonl --out demo.html
ccb version

ccb certify and ccb leaderboard accept --json and print exactly one ccb/v1 envelope. Schemas live in schema/ and ship inside the wheel.

Input shape

One record per line:

{"model_a":"model-x","model_b":"model-y","outcome":1.0,"t":0}

outcome is 1.0 if model_a wins, 0.0 if model_b wins, 0.5 for a tie. t and variant are optional; --format csv reads the same columns from a CSV header.

Limitations

  • A certificate is robustness to corrupted records, not proof that a model is better, and not fraud detection.
  • ccb replay estimates false-statement rates by resampling the input stream; the rates are meaningful only when the uncorrupted claim is false.
  • A leaderboard is a snapshot at the shortest input stream. There is no streaming update of a published leaderboard.
  • A -1 budget means no certificate was issued at the chosen alpha; it is not evidence against the claim.
  • The guarantee assumes alpha, the horizon, the corruption cost, the pair set, and the variant count were fixed before the data was seen. Choices made after looking at the data need their own correction.
  • Local files only. There is no hosted ingestion, account, or database.
  • The paper is a preprint; the implementation is not peer reviewed.

Security and contributing

Report vulnerabilities through a GitHub security advisory, not a public issue.

Development: uv sync, then uv run pytest and uv run ruff check .. A change to a certificate or leaderboard formula needs a proof or a citation to the paper; name the section in the function docstring.

Licence

MIT (see LICENSE).

Metadata

Release files for certified-corruption-budget 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for certified-corruption-budget 0.1.0
File Size Uploaded Provenance Actions
certified_corruption_budget-0.1.0.tar.gz 17.1 kB Without provenance Details for certified_corruption_budget-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for certified-corruption-budget 0.1.0
File Interpreter ABI Platform Actions
certified_corruption_budget-0.1.0-py3-none-any.whl Python 3 none any Details for certified_corruption_budget-0.1.0-py3-none-any.whl

Total release size: 39.6 kB

Release files / certified_corruption_budget-0.1.0.tar.gz

Download URL certified_corruption_budget-0.1.0.tar.gz
Size 17.1 kB
Tags Source
SHA-256 checksum
How to use checksums
1c02fdd1266da2fb4097ee3b9deb3417d18869be0021a3f81b7d88536f42c7e4
BLAKE2b-256 checksum
How to use checksums
17832c67c8ac61804575b9de48afbb894501dc0fc7efa403223e8e88d26091ac
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / certified_corruption_budget-0.1.0-py3-none-any.whl

Download URL certified_corruption_budget-0.1.0-py3-none-any.whl
Size 22.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f0607a04ff05659ffc617a692c3887cfe1c53e64785f8195783f930b64f1720f
BLAKE2b-256 checksum
How to use checksums
d7de2a70c0b1ba4817e5c14177f2736d0eeab5e901e18f7c3eab13684d7d727d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for certified-corruption-budget 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page