RIDI

Allocation identity for capacity-limited AI

See what aggregate evaluation can miss: who actually received the scarce slots.

PyPI · Open in Colab · 60-second Quick Start · Python API · Reporting checklist · Paper & evidence

The one-line idea

Performance, fairness, calibration and ranking metrics remain necessary. RIDI asks a different question at the score-to-action boundary:

Which identities occupy the finite queue, shortlist or context window—and did those identities change?

Two systems can be exactly indistinguishable under a declared audit while assigning scarce slots to different identities. In the preregistered RAG experiment behind this project, that difference sometimes changed the downstream AI decision itself.

Install and use it in seconds

pip install ridi-audit
ridi-audit demo

Already have two selected lists?

from ridi_audit import compare_allocations

reference = ["doc-1", "doc-2", "doc-3", "doc-4"]
updated   = ["doc-1", "doc-2", "doc-9", "doc-4"]

report = compare_allocations(reference, updated)
print(report)

RIDI Allocation Comparison
--------------------------
Before size:   4
After size:    4
Overlap:       3
Changed slots: 1
RIDI:          0.400000

This is the shortest path for RAG contexts, shortlists, alert queues, remediation lists and other realized top-k allocations.

Have paired candidate scores?

import pandas as pd
from ridi_audit import audit

before = pd.read_csv("before.csv")
after = pd.read_csv("after.csv")

report = audit(before, after, k=[10, 50, 100])
controlled = report.control(k=100, eta=0.001)

What RIDI adds

For equal-size selected sets A and B:

RIDI(A, B) = 1 - |A ∩ B| / |A ∪ B|

RIDI is not a replacement for AUROC, precision, recall, nDCG, calibration, robustness, safety or fairness. It makes realized membership observable.

Evidence behind the project

Most RAG benchmark-defined outcomes remained stable. The claim is behavioral non-equivalence can exist inside an exactly audit-equivalent class, not that every allocation change is harmful or that all AI systems are fragile.

Independent verification status — 4 Sep 2026

• The sealed EPSS numerical workflow was reproduced by two independent external executors in separate environments.
• SciFact 275 was independently regenerated twice, blind, by external executors using distinct serving environments.
• Mohammed Hamdan: all registered retrieval-audit quantities remained identical; reference and identity control were SUPPORTS (the identity output was byte-identical to reference), the order-only permutation remained SUPPORTS, and the audit-equivalent identity substitution produced REFUTES. His run used the same frozen inputs and prompt conditioning but, because of hardware limits, used a disclosed Q4_K_M qwen3:8b llama.cpp/Ollama serving path rather than the frozen bf16 Hugging Face pipeline.
• Théophile Ossard: independently reproduced the same substantive SUPPORTS versus REFUTES pattern on a distinct GPU/software stack. His regenerated reference prefixed the verdict with Verdict:, exposing a strict-parser boundary that is retained transparently.
• These are independent computational regenerations, not a CODECHECK certificate. Community CODECHECK request #208 is registered; formal checking has not begun.

Read the evidence and boundaries →

Designed to drop into research workflows

Framework-agnostic. RIDI needs identities—and optionally scores—not a particular model library.

Two entry points. compare_allocations() for selected IDs; audit() for paired score tables.

Publication-ready output. Reports can emit dictionaries or Markdown records.

Control, not only measurement. AuditReport.control() exposes the exact identity–utility frontier.

Reproducible by default. Deterministic tie handling, CI on Python 3.10–3.12, locked research workflows, negative results retained, and explicit verification boundaries.

Common settings: RAG and information retrieval · cybersecurity remediation · clinical alerts · fraud/compliance triage · inspection queues · hiring/shortlisting · grant review · moderation · any budget-constrained ranking or finite action set.

Copy-paste recipes →

Use it in a paper

The Allocation Identity Reporting Checklist covers capacity, selection rule, comparator, identity metrics, controls, conventional evaluation, privacy and downstream outcomes.

A minimal methods sentence is:

We audited allocation identity at the prespecified capacity by reporting selected-set overlap, changed slots and RIDI alongside the domain’s conventional evaluation metrics.

Adapt it to the design; do not claim endpoints you did not test.

Replicate it. Challenge it. Extend it.

Independent reproductions, boundary cases, discrepancies and null results are welcome. Use the dedicated GitHub issue forms for an Independent replication or a New domain application.

Contributing guide →

Current manuscript

Identical audits can yield different AI decisions

The public repository is synchronized to the current science/product lock as of 4 Sep 2026. The manuscript is prepared for journal submission; it is not peer reviewed, accepted or published.

Registered RxNorm and Open Targets failures remain part of the project record to delimit the claim. Allocation identity alone does not establish correctness, fairness, causal harm, benefit, clinical utility or model superiority.

• Paper overview
• RAG preregistration
• Reproducibility guide
• Numerical provenance
• CODECHECK request #208

Citation

If you use RIDI or ridi-audit, cite the software using CITATION.cff and the accompanying manuscript when a public bibliographic record is available.

Adeeb Noor
Department of Information Technology, Faculty of Computing and Information Technology
King Abdulaziz University, Jeddah, Saudi Arabia
ORCID: 0000-0002-8251-1853