xai-auditor

Quantitative Explainability Auditing and Model Risk Management for Machine Learning.

xai-auditor is an independent, production-grade Python library designed to audit the reliability, stability, fidelity, consistency, manipulation resistance, distributional drift, and operational coverage of post-hoc Explainable AI (XAI) pipelines such as SHAP and LIME.

Author: Fdere AI, Explainability Auditor Team
License: Apache License 2.0
Current Version: 0.2.1

---

Table of Contents

1. Project Overview
2. Key Capabilities
3. Why Explainability Auditing Matters
4. Auditing Architecture
5. Six Audit Dimensions
6. DriftAuditor v0.2.0 Redesign
7. Methodology and Scoring Mathematics
8. Installation
9. Quick Start
10. Real-World Dataset Examples
11. Full Governance Audit Example
12. Interpreting Results
13. Configuration
14. Backward Compatibility
15. Validation and Benchmark Results
16. Testing
17. Project Structure
18. Limitations
19. Future Work
20. Credits
21. License

---

Project Overview

In regulated financial services, clinical healthcare, and high-consequence algorithmic decision-making, post-hoc explanations generated by surrogate methods such as SHAP and LIME are often relied upon for adverse action notices, compliance reporting, and clinician validation. However, surrogate explanations are themselves stochastic estimations that can suffer from numerical instability, cross-method disagreement, adversarial manipulation, and distributional drift over time.

xai-auditor provides a quantitative verification harness that evaluates surrogate explainers with mathematical rigor. It assesses explanation pipelines across six failure modes, aggregates empirical metrics into a composite Explainability Trust Score (0 to 100), and issues structured findings with actionable recommendations.

---

Key Capabilities

• Universal Model Compatibility: Audits Scikit-Learn estimators, tree ensembles (XGBoost, LightGBM, CatBoost), neural networks (PyTorch, TensorFlow), built-in reference estimators, or arbitrary prediction callables f(X) -> probabilities.
• Built-in Explainers: Includes Permutation SHAP with efficiency (additivity) constraints and Kernel LIME with local ridge surrogates, requiring zero external explainer dependencies.
• Six Independent Audit Dimensions: Evaluates attribution fidelity, local Lipschitz stability, cross-method consistency, adversarial manipulation resistance, distributional drift, and operational coverage.
• Calibrated Scoring and Governance Risk Tiers: Maps continuous empirical measurements into standardized risk classifications (Low Risk, Medium Risk, High Risk, Critical Risk).
• Multi-Format Reporting: Generates machine-readable JSON audit certificates, human-readable CLI summaries, and self-contained HTML assurance reports.
• Zero Required External Dependencies: Pure Python implementation of linear algebra, optimization, quantile calculations, and statistical distances.

---

Why Explainability Auditing Matters

Machine learning models deployed in production environments are governed by internal model risk management policies and regulatory frameworks such as US Federal Reserve SR 11-7 (Guidance on Model Risk Management) and EU AI Act Article 13 (Transparency and Provision of Information).

Surrogate explanations present distinct operational and governance risks:

1. Surrogate Unfaithfulness: Explanations can point to features that have minimal causal impact on the underlying model's predictions.
2. Local Instability: Tiny, non-semantic perturbations in input features can cause dramatic shifts in feature importance rankings.
3. Cross-Method Disagreement: SHAP and LIME frequently disagree on the top decision drivers for the exact same inference instance.
4. Hyperparameter Cherry-Picking: An operator can manipulate perceived feature importance by altering surrogate kernel bandwidths or background reference sets.
5. Silent Attribution Drift: Covariate or concept shifts can alter the distribution of model explanations between training and production inference, even when top-line accuracy appears stable.
6. Production Latency and Failures: Surrogate methods with high computational complexity can breach real-time service level agreements (SLAs) or fail silently.

xai-auditor provides an objective, automated mechanism to detect, quantify, and report these failure modes.

---

Auditing Architecture

The xai-auditor engine is structured into five modular layers:

+-------------------------------------------------------------------------+
|                           Auditor Pipeline                              |
|                 (xai_auditor.Auditor / xai-audit CLI)                   |
+-------------------------------------------------------------------------+
       |                  |                  |                  |
+-------------+    +-------------+    +-------------+    +-------------+
|    Model    |    |  Surrogate  |    | Preprocess  |    |  Dimension  |
|  Adapters   |    | Explainers  |    |  Pipeline   |    |   Runners   |
| (base.py,   |    | (shap.py,   |    | (validation |    | (fidelity,  |
| builtins.py)|    |  lime.py)   |    |  split.py)  |    |  drift,...) |
+-------------+    +-------------+    +-------------+    +-------------+
       \                  /                  \                  /
        +----------------+--------------------+----------------+
                                 |
+-------------------------------------------------------------------------+
|                             Scoring Engine                              |
|          (calculate_trust_score, classify_risk, dimension_scores)       |
+-------------------------------------------------------------------------+
                                 |
+-------------------------------------------------------------------------+
|                          Reporting & Artifacts                          |
|         (AuditResult, JSON Certificates, HTML Assurance Reports)        |
+-------------------------------------------------------------------------+

1. Model Adapters (xai_auditor.models): Universal interfaces standardizing probability outputs across custom prediction callables, Scikit-Learn models, and built-in estimators.
2. Surrogate Explainers (xai_auditor.explainers): Self-contained implementations of cooperative game-theoretic SHAP and local linear LIME.
3. Dimension Auditors (xai_auditor.auditors): Six specialized test runners executing targeted perturbation, ablation, consistency, and drift experiments.
4. Scoring Engine (xai_auditor.scoring): Deterministic functions converting raw metric values to the [0, 100] regulatory score spectrum.
5. Results and Reporting (xai_auditor.results, xai_auditor.reporting): Structured data models capturing metrics, findings, execution timings, and export functions.

---

Six Audit Dimensions

The composite Explainability Trust Score (0 to 100) aggregates all six dimensions using established governance weights:

Audit Dimension	Weight	Primary Testing Objective	Key Metrics Evaluated
Attribution Fidelity	25%	Causal accuracy of feature attribution under systematic feature removal and retention.	ROAR Effectiveness, KAR Recovery Rate
Local Lipschitz Stability	20%	Invariance of attribution rankings under small epsilon-Gaussian feature perturbations.	Spearman Rank Correlation, Local Consistency, Attribution Variance
Cross-Method Consistency	15%	Multi-explainer consensus between SHAP and LIME on individual instances.	Top-K Jaccard Similarity, Sign Agreement, Rank-Biased Overlap (RBO), Cosine Similarity
Adversarial Manipulation Resistance	15%	Robustness of explanations against explainer hyperparameter variations (kernel widths, background sizes).	Explanation Manipulation Index (EMI)
Distributional Explanation Drift	15%	Temporal and cohort shift between baseline and monitoring attribution distributions.	Mean Quantile PSI, Standardized Wasserstein Distance
Operational Coverage & Latency	10%	Reliability of explanation generation and compliance with production latency SLAs.	Success Rate (%), p50 / p95 / p99 Latency (ms)

---

DriftAuditor v0.2.0 Redesign

Problem Formulation in v0.1.0

In xai-auditor v0.1.0, the DriftAuditor evaluated distribution divergence using equal-width histogram binning and aggregated results via the maximum feature Population Stability Index ($\max_j \text{PSI}_j$).

Experimental benchmarking revealed fundamental failure modes under this legacy design:

1. Sparse Equal-Width Bins: Post-hoc attributions typically follow heavy-tailed distributions with extreme density concentrated near zero. Equal-width binning created many empty or near-empty bins in the tails, causing severe probability estimation instability and $\ln(q/p)$ numerical explosion.
2. Extreme PSI Inflation: Even on independent and identically distributed (IID) validation splits with no true underlying drift, small sample fluctuations in sparse tail bins inflated PSI values past 0.25, triggering false-positive alerts.
3. Single-Feature Max PSI Domination: In high-dimensional datasets ($p \ge 10$), evaluating $\max_j \text{PSI}_j$ meant that noise in a single uninformative feature was sufficient to fail the entire audit.
4. Score-Floor Collapse: Because any PSI exceeding 0.25 incurred maximum penalty, the legacy scoring function collapsed to the floor score of 35 on 100% of IID benchmark runs across Banknote Authentication, Statlog Heart Disease, and WDBC.
5. Raw Wasserstein Scale Sensitivity: The legacy 1D Wasserstein distance was unstandardized, making it dependent on feature scale and model coefficient magnitudes.
6. Small-Window Instability: A small default window ($N = 20$) generated high sampling variance.

Methodological Solution in v0.2.0

To resolve these issues, v0.2.0 introduced a redesigned methodology:

1. Baseline-Anchored Empirical Quantile Binning: Quantile cutpoints $e_0, e_1, \dots, e_B$ are derived from the baseline distribution such that each interval contains equal baseline mass ($1/B$). By default, $B = 4$ quartiles are used.
2. Duplicate Quantile Handling and Outer Support Expansion: When feature attributions contain discrete spikes (e.g. repeated zeros), tied quantiles are deduplicated. Outer boundaries are expanded to encompass the full range of both baseline and monitoring samples: $$e_0 = \min(e_0, \min(X_{\text{base}}), \min(X_{\text{mon}})) - 10^{-9}, \quad e_B = \max(e_B, \max(X_{\text{base}}), \max(X_{\text{mon}})) + 10^{-9}$$ This guarantees that 100% of monitoring observations are captured without bin clipping.
3. Mean Quantile PSI Aggregation: Rather than relying on the noisy maximum, the dimension-wide average $\overline{\text{PSI}} = \frac{1}{p} \sum_{j=1}^{p} \text{PSI}_j$ serves as the primary PSI signal.
4. Standardized 1D Wasserstein Distance: Continuous distribution shift is measured using the closed-form 1D Earth Mover's Distance normalized by baseline standard deviation ($\sigma_{\text{base}, j}$), making it scale-invariant: $$W_{1,\text{std}, j} = \frac{W_1(P_j, Q_j)}{\sigma_{\text{base}, j} + 10^{-8}}, \quad \overline{W}{1,\text{std}} = \frac{1}{p} \sum{j=1}^{p} W_{1,\text{std}, j}$$
5. Dynamic Window Sizing: The default target sample window was increased to $N = 100$. For small datasets where $N < 200$, window size scales dynamically to $\min(\lfloor N_{\text{available}} / 2 \rfloor, 100)$, preventing partition collapse.
6. Calibrated Hybrid Scoring: The drift score combines non-parametric frequency shifts (60% weight on PSI) and optimal transport distance (40% weight on Standardized Wasserstein) into a monotonic score mapped to the [35, 98] interval.

---

Methodology and Scoring Mathematics

Quantile Population Stability Index (PSI)

For each feature $j \in {1, \dots, p}$, the baseline sample is sorted to obtain empirical quartile edges $e_0, e_1, e_2, e_3, e_4$. Frequencies of baseline and monitoring attributions falling within each bin $[e_{b-1}, e_b)$ are calculated:

$$\hat{p}b = \frac{1}{N{\text{base}}} \sum_{i=1}^{N_{\text{base}}} \mathbb{I}(a_{\text{base}, i} \in \text{bin}b), \quad \hat{q}b = \frac{1}{N{\text{mon}}} \sum{i=1}^{N_{\text{mon}}} \mathbb{I}(a_{\text{mon}, i} \in \text{bin}_b)$$

Applying epsilon smoothing ($\epsilon = 10^{-4}$) to prevent zero denominators:

$$p_b = \max(\epsilon, \hat{p}_b), \quad q_b = \max(\epsilon, \hat{q}_b)$$

$$\text{PSI}j = \sum{b=1}^{B} (q_b - p_b) \ln\left(\frac{q_b}{p_b}\right)$$

Standardized Wasserstein Distance

For 1D empirical distributions, the Wasserstein distance equals the integrated difference between quantile functions (empirical inverse cumulative distribution functions):

$$W_1(P_j, Q_j) = \frac{1}{K} \sum_{k=1}^{K} |F_{P_j}^{-1}(t_k) - F_{Q_j}^{-1}(t_k)|$$

Standardizing by baseline standard deviation $\sigma_{\text{base}, j} = \sqrt{\frac{1}{N-1} \sum (a_{\text{base}, i} - \bar{a}_{\text{base}})^2}$:

$$W_{1,\text{std}, j} = \frac{W_1(P_j, Q_j)}{\sigma_{\text{base}, j} + 10^{-8}}$$

If baseline variance is zero ($\sigma_{\text{base}, j} < 10^{-8}$), the function returns 0.0 if $W_1 < 10^{-8}$, or caps at 10.0 if distributions differ.

Hybrid Drift Score Formulation

$$\text{Component}{\text{PSI}} = \min(1.0, 2.0 \cdot \overline{\text{PSI}})$$ $$\text{Component}{\text{Wass}} = \min(1.0, 0.5 \cdot \overline{W}{1,\text{std}})$$ $$\text{Penalty} = \min(1.0, 0.6 \cdot \text{Component}{\text{PSI}} + 0.4 \cdot \text{Component}_{\text{Wass}})$$ $$\text{Score} = \max\left(35, \min\left(98, \text{round}\left(98.0 - 63.0 \cdot \text{Penalty}\right)\right)\right)$$

Overall Composite Trust Score

The final composite Trust Score aggregates the scores of all active dimensions $D_k$ using their configured weights $w_k$:

$$\text{Trust Score} = \frac{\sum_{k} w_k \cdot \text{Score}(D_k)}{\sum_{k} w_k}$$

---

Installation

xai-auditor is completely self-contained with no required external dependencies for its core mathematical operations.

# Standard installation
pip install xai-auditor

# Optional installation with integrations for scikit-learn, xgboost, lightgbm
pip install xai-auditor[all]

Requirements

• Python 3.8, 3.9, 3.10, 3.11, or 3.12
• Operating System: Linux, macOS, or Windows

---

Quick Start

from xai_auditor import Auditor
from xai_auditor.models.builtins import BuiltinLogisticRegression

# 1. Prepare sample data
X = [
    [0.85, 1.2, 0.45, 0.90],
    [0.20, 0.4, 0.10, 0.25],
    [0.70, 1.0, 0.60, 0.75],
    [0.15, 0.3, 0.05, 0.10],
    [0.90, 1.5, 0.80, 0.95],
    [0.30, 0.5, 0.20, 0.35],
]
y = [1, 0, 1, 0, 1, 0]
feature_names = ["debt_to_income", "credit_lines", "delinquency", "utilization"]

# 2. Fit a built-in regularized model
model = BuiltinLogisticRegression().fit(X, y)

# 3. Instantiate and run the auditor
auditor = Auditor(
    model=model,
    X=X,
    y=y,
    feature_names=feature_names,
    dataset_name="Credit Underwriting Demo",
)
result = auditor.run()

# 4. View results
print(f"Trust Score:      {result.trust_score} / 100")
print(f"Risk Tier:        {result.risk_tier.value}")
print(f"Approval Status:  {result.approval_status.value}")

---

Real-World Dataset Examples

Example 1: Standalone DriftAuditor (UCI Statlog Heart Disease)

The Statlog Heart Disease dataset comprises clinical attributes (e.g. resting blood pressure, serum cholesterol, ST depression) used to predict heart disease presence. In clinical deployments, patient cohort demographics and comorbidities shift across facilities.

This example demonstrates auditing explanation drift between a baseline historical population and a monitoring cohort experiencing moderate clinical shift:

import random
from xai_auditor import DriftAuditor
from xai_auditor.models.builtins import BuiltinLogisticRegression

# 1. Feature definitions (13 clinical indicators)
feature_names = [
    "age", "sex", "chest_pain_type", "resting_bp", "cholesterol",
    "fasting_blood_sugar", "resting_ecg", "max_heart_rate",
    "exercise_angina", "oldpeak", "slope", "num_vessels", "thal"
]

# Clinical risk model coefficients
weights = [0.4, 1.2, 0.8, 0.3, 0.2, 0.1, 0.3, -0.9, 0.7, 1.1, 0.6, 1.4, 0.9]
model = BuiltinLogisticRegression(weights=weights, bias=0.0)

# 2. Baseline population (historical clinical trial cohort, N=100)
rng = random.Random(42)
X_baseline = [
    [rng.gauss(0.0, 1.0) for _ in range(len(feature_names))]
    for _ in range(100)
]

# 3. Monitoring population (inpatient cohort with elevated cardiac stress markers)
# Shifting oldpeak (idx 9), slope (idx 10), and num_vessels (idx 11) by 1.0 sigma
X_monitoring = [
    [
        val + (1.0 if j in (9, 10, 11) else 0.0)
        for j, val in enumerate([rng.gauss(0.0, 1.0) for _ in range(len(feature_names))])
    ]
    for _ in range(100)
]

# 4. Configure DriftAuditor (v0.2.0 defaults: 4 quantile bins, target window 100)
drift_auditor = DriftAuditor(
    n_samples_per_window=100,
    bin_strategy="quantile",
    n_bins=4,
    seed=42,
)

# 5. Execute drift audit
dim_result = drift_auditor.audit(
    model=model,
    X=X_baseline,
    X_monitoring=X_monitoring,
    feature_names=feature_names,
    explainer_type="shap",
)

# 6. Inspect drift metrics and score
print(f"Dimension:                     {dim_result.name}")
print(f"Drift Score:                   {dim_result.score} / 100")
print(f"Mean Quantile PSI:             {dim_result.metrics['mean_feature_psi']:.4f}")
print(f"Mean Standardized Wasserstein: {dim_result.metrics['mean_standardized_wasserstein']:.4f}")
print(f"Legacy Max Feature PSI:        {dim_result.metrics['max_feature_psi']:.4f}")
print(f"Baseline Samples:              {dim_result.metrics['baseline_samples']}")
print(f"Monitoring Samples:            {dim_result.metrics['monitoring_samples']}")

# 7. Print findings
for finding in dim_result.findings:
    print(f"\nFinding [{finding.severity}]: {finding.title}")
    print(f"Details: {finding.description}")
    print(f"Action:  {finding.recommendation}")

Example 2: UCI Banknote Authentication (Stationary IID Verification)

The Banknote Authentication dataset uses continuous wavelet transform features (variance, skewness, curtosis, entropy) from image captures. This example demonstrates verifying that stationary IID populations correctly pass the audit without false-positive drift alarms:

import random
from xai_auditor import DriftAuditor
from xai_auditor.models.builtins import BuiltinLogisticRegression

banknote_features = ["variance_wavelet", "skewness_wavelet", "curtosis_wavelet", "entropy_wavelet"]
model = BuiltinLogisticRegression(weights=[1.5, -2.1, 1.8, -0.4], bias=0.1)

# Generate stationary IID data split
rng = random.Random(101)
all_samples = [[rng.gauss(0.0, 1.0) for _ in range(4)] for _ in range(200)]
X_baseline = all_samples[:100]
X_monitoring = all_samples[100:]

drift_auditor = DriftAuditor(n_samples_per_window=100, bin_strategy="quantile", seed=101)
result = drift_auditor.audit(model=model, X=X_baseline, X_monitoring=X_monitoring, feature_names=banknote_features)

print(f"Banknote IID Drift Score: {result.score} / 100 (Expected: >= 85 Low Risk / PASS)")
print(f"Mean Quantile PSI:        {result.metrics['mean_feature_psi']:.4f}")
print(f"Mean Std Wasserstein:     {result.metrics['mean_standardized_wasserstein']:.4f}")

---

Full Governance Audit Example

Wisconsin Diagnostic Breast Cancer (WDBC) 6-Dimension Audit

The WDBC dataset contains 30 features derived from digitized images of fine needle aspirate (FNA) biopsies. The following example demonstrates executing the complete 6-dimension governance audit:

import random
from xai_auditor import Auditor
from xai_auditor.models.builtins import BuiltinRandomForest

# 1. Define 30 diagnostic cell nuclei features
p = 30
wdbc_features = [f"cell_feature_{i+1}" for i in range(p)]

# 2. Prepare synthetic diagnostic dataset
rng = random.Random(2024)
X_wdbc = [[rng.gauss(0.0, 1.0) for _ in range(p)] for _ in range(120)]
y_wdbc = [1 if (row[0] * 1.5 + row[1] * 0.8 + row[2] * -1.2) > 0.0 else 0 for row in X_wdbc]

# 3. Fit Random Forest classifier
classifier = BuiltinRandomForest(n_estimators=15, max_depth=4, seed=42).fit(X_wdbc, y_wdbc)

# 4. Instantiate full governance auditor
auditor = Auditor(
    model=classifier,
    X=X_wdbc,
    y=y_wdbc,
    feature_names=wdbc_features,
    dataset_name="WDBC Diagnostic Pathology Suite",
    model_name="Malignancy Prediction RF v2",
    explainer_type="shap",
    seed=42,
    drift_kwargs={"n_samples_per_window": 60, "bin_strategy": "quantile", "n_bins": 4},
    fidelity_kwargs={"max_ablation_steps": 5},
    stability_kwargs={"epsilon": 0.03},
    consistency_kwargs={"top_k": 5, "rbo_p": 0.85},
    manipulation_kwargs={"n_samples": 8},
    coverage_kwargs={"batch_size": 15},
)

# 5. Run audit and export report
audit_result = auditor.run()

print("=" * 60)
print(f"Model:            {audit_result.model_name}")
print(f"Dataset:          {audit_result.dataset_name}")
print(f"Trust Score:      {audit_result.trust_score} / 100")
print(f"Risk Tier:        {audit_result.risk_tier.value}")
print(f"Approval Status:  {audit_result.approval_status.value}")
print("=" * 60)

for dim in audit_result.dimension_results:
    print(f"{dim.name:<40} Score: {dim.score:>3} / 100  (Weight: {dim.weight*100:.0f}%)")

# Save self-contained HTML report
audit_result.save_html_report("wdbc_governance_report.html")
print("\nHTML assurance certificate saved to wdbc_governance_report.html")

---

Interpreting Results

Regulatory Risk Tiers

xai-auditor classifies the composite Trust Score into four standardized risk tiers:

Risk Tier	Score Range	Approval Determination	Action Required
Low Risk	85 to 100	APPROVED	Unconditional approval for production deployment. Explanations exhibit high fidelity, stability, consistency, and stationary distributions.
Medium Risk	75 to 84	APPROVED	Approved with standard production telemetry and scheduled re-auditing. Minor non-critical discrepancies observed.
High Risk	60 to 74	CONDITIONAL	Conditional approval; requires model risk committee review and enhanced monitoring. Explanations show notable instability or drift.
Critical Risk	0 to 59	REJECTED	Production deployment prohibited. Mandatory model or explainer remediation required before reconsideration.

Drift Dimension Metric Guide

• Mean Quantile PSI (mean_feature_psi):
  • $< 0.10$: Stationary. No statistically significant attribution distribution divergence.
  • $0.10 \text{ to } 0.25$: Moderate drift. Explanations indicate slight shifts in feature importance distributions.
  • $> 0.25$: Significant drift. Substantial divergence in attribution profiles between baseline and monitoring cohorts.
• Mean Standardized Wasserstein (mean_standardized_wasserstein):
  • $< 0.30$: Minimal distribution transport distance.
  • $0.30 \text{ to } 0.80$: Moderate distribution shift (0.3 to 0.8 baseline standard deviations).
  • $> 1.00$: Substantial distribution shift (>1.0 baseline standard deviation).

---

Configuration

DriftAuditor Parameters

from xai_auditor import DriftAuditor

drift_auditor = DriftAuditor(
    split_ratio=0.5,             # Train/test split ratio when X_monitoring is not provided
    n_samples_per_window=100,    # Target sample size per evaluation partition
    bin_strategy="quantile",     # "quantile" (v0.2.0 default) or "equal_width" (v0.1.0 legacy)
    n_bins=4,                    # Number of quantile bins (default: 4 quartiles)
    seed=42,                     # PRNG seed for reproducible sample splitting
)

Full Auditor Parameters

from xai_auditor import Auditor

auditor = Auditor(
    model=model,                 # Model instance, adapter, or callable f(X) -> probabilities
    X=X,                         # Baseline feature matrix (2D list of floats)
    y=y,                         # Ground-truth binary classification targets (optional)
    feature_names=feature_names, # List of string feature names
    dataset_name="Credit Data",  # Identifier for governance reporting
    model_name="Risk Model v1",  # Identifier for governance reporting
    explainer_type="shap",       # "shap" or "lime"
    seed=42,                     # Top-level PRNG seed (propagated to all sub-auditors)
    dimension_weights={          # Custom dimension weights (must sum to 1.0)
        "fidelity": 0.25,
        "stability": 0.20,
        "consistency": 0.15,
        "manipulation": 0.15,
        "drift": 0.15,
        "coverage": 0.10,
    },
    drift_kwargs={"n_samples_per_window": 100, "bin_strategy": "quantile", "n_bins": 4},
    fidelity_kwargs={"n_samples": 15, "max_ablation_steps": 6},
    stability_kwargs={"epsilon": 0.02, "n_samples": 15},
    consistency_kwargs={"top_k": 3, "rbo_p": 0.9, "n_samples": 15},
    manipulation_kwargs={"n_samples": 10},
    coverage_kwargs={"batch_size": 20},
)

---

Backward Compatibility

xai-auditor v0.2.0 maintains complete backward compatibility with v0.1.0 pipelines:

1. Constructor Signatures: DriftAuditor supports all legacy arguments (split_ratio, n_samples_per_window, bin_strategy, n_bins, seed).
2. Equal-Width Binning Mode: Passing bin_strategy="equal_width" preserves v0.1.0 uniform histogram binning.
3. Metric Keys in DimensionResult.metrics: The key "max_feature_psi" is retained alongside "mean_feature_psi", "mean_wasserstein_distance", "mean_standardized_wasserstein", "baseline_samples", and "monitoring_samples".
4. Scoring Function: Calling score_drift(max_feature_psi=...) continues to function for legacy callers.
5. Public API and CLI: All public module exports in xai_auditor.__all__ and the xai-audit command-line entry point remain unchanged.

---

Validation and Benchmark Results

The v0.2.0 DriftAuditor redesign was validated through 30-run Monte Carlo experiments across three standard benchmark datasets under four controlled drift conditions:

• Stationary / IID: Independent and identically distributed splits with no true underlying drift.
• Mild Drift: $0.5\sigma$ mean shift applied to 25% of features.
• Moderate Drift: $1.0\sigma$ mean shift applied to 50% of features.
• Strong Drift: $2.0\sigma$ mean shift applied to 100% of features.

Benchmark Summary Table

Dataset	Dimensionality ($p$)	v0.1.0 IID False Positive Rate	v0.2.0 IID False Positive Rate	v0.2.0 Mean IID Score (95% CI)	v0.2.0 Floor Hits (Score=35)
Banknote Authentication	$p = 4$	100.0% (30/30)	20.0% (6/30)	$88.37 \pm 4.73$ [86.67, 90.06]	3/120 (2.5%)
Statlog Heart Disease	$p = 13$	100.0% (30/30)	6.7% (2/30)	$90.23 \pm 3.85$ [88.86, 91.61]	0/120 (0.0%)
WDBC Breast Cancer	$p = 30$	100.0% (30/30)	10.0% (3/30)	$89.77 \pm 4.19$ [88.27, 91.27]	9/120 (7.5%)*

*In v0.2.0, floor hits (score = 35) occurred exclusively under the True Strong Drift condition ($2.0\sigma$ shift across 100% of features), confirming proper discriminative saturation rather than artificial floor collapse.

Key Validation Takeaways

1. False Positive Suppression: False positive rates under stationary conditions dropped from 100% across all datasets in v0.1.0 to 6.7% to 20.0% in v0.2.0.
2. Elimination of Floor Collapse: The artificial collapse where stationary distributions received the minimum possible score of 35 was eliminated.
3. Monotonic Ordering Restored: Score progression strictly follows the expected ordering: $\text{Score}(\text{IID}) > \text{Score}(\text{Mild}) > \text{Score}(\text{Moderate}) > \text{Score}(\text{Strong})$.
4. Scale Invariance Verified: Standardized Wasserstein distance was verified to be strictly invariant under $10\times$ and $100\times$ feature scale multiplications.

---

Testing

xai-auditor includes a comprehensive test suite of 64 automated tests executed via Python's standard unittest framework.

Running Tests

# Run complete test suite
python3 -m unittest discover tests -p "test_*.py" -v

# Run dedicated DriftAuditor redesign tests
python3 -m unittest tests/test_drift_redesign.py -v

Test Suite Coverage

• test_drift_redesign.py (10 tests): Verifies quantile binning accuracy, equal-width backward compatibility, tied quantile deduplication, constant/zero-variance feature handling, small sample dynamic windowing, Wasserstein scale invariance, hybrid scoring calibration, IID robustness, outlier resistance, and end-to-end Auditor integration.
• test_audit_dimensions.py (6 tests): Verifies individual execution of all six dimension auditors.
• test_deep_verification.py (7 tests): Verifies all built-in models, weight normalization, risk tier boundaries, PSI edge cases, RBO ranking properties, and ROAR/KAR curves.
• test_explainers.py (3 tests): Verifies Permutation SHAP efficiency/additivity and Kernel LIME local ridge surrogates.
• test_linalg_and_metrics.py (6 tests): Verifies matrix solvers, correlation metrics, sign agreement, and vector operations.
• test_public_api.py (4 tests): Verifies Auditor pipeline, single-dimension execution, JSON serialization, and HTML report export.
• test_ranking_and_rbo.py (6 tests): Verifies Kendall Tau, Jaccard similarity, and RBO convergence properties.
• test_regressions.py (6 tests): Verifies regression fixes including UniversalModelAdapter model_name setter and Auditor seed propagation.
• test_scoring_and_risk.py (3 tests): Verifies risk threshold classification and trust score calculation math.
• test_validation.py (7 tests): Verifies input dataset validation, cardinality checks, zero-variance checks, and missing value imputation.
• test_cli.py and test_edge_cases.py (6 tests): Verifies CLI execution and custom callable adapters.

---

Project Structure

xai-auditor/
├── pyproject.toml              # Build metadata (v0.2.0)
├── setup.py                    # Setuptools build script (v0.2.0)
├── setup.cfg                   # Packaging configuration (v0.2.0)
├── README.md                   # Technical documentation
├── examples/                   # Runnable usage examples
│   ├── basic_audit.py          # Quick-start audit script
│   ├── custom_model_audit.py   # Auditing custom prediction callables
│   └── dimension_specific_audit.py # Auditing specific individual dimensions
├── src/
│   └── xai_auditor/
│       ├── __init__.py         # Public API exports
│       ├── __version__.py      # Version declaration (0.2.0)
│       ├── auditor.py          # Main Auditor orchestration engine
│       ├── auditors/           # Six dimension auditor implementations
│       │   ├── base.py
│       │   ├── consistency.py
│       │   ├── coverage.py
│       │   ├── drift.py        # DriftAuditor v0.2.0 implementation
│       │   ├── fidelity.py
│       │   ├── manipulation.py
│       │   ├── runner.py
│       │   └── stability.py
│       ├── cli/                # Command-line interface (xai-audit)
│       ├── core/               # Pure Python linalg, statistics, preprocessing
│       ├── explainers/         # Permutation SHAP and Kernel LIME surrogates
│       ├── metrics/            # Mathematical metric implementations (PSI, Wasserstein, RBO)
│       ├── models/             # Built-in models and universal adapters
│       ├── reporting/          # HTML report generation
│       ├── results/            # Data structures (AuditResult, DimensionResult, Finding)
│       └── scoring/            # Dimension and trust scoring algorithms
└── tests/                      # Automated test suite (64 tests)

---

Limitations

1. Marginal 1D Distribution Analysis: DriftAuditor currently evaluates marginal attribution drift per feature independently. Joint multidimensional attribution shifts (such as cross-feature interaction drift) are not captured by 1D PSI or 1D Wasserstein.
2. Unweighted Feature Aggregation in Mean PSI: Features are averaged with equal weight ($1/p$). In models with sparse feature usage, uninformative features can dilute significant drift in core drivers.
3. Surrogate Approximation Error: Permutation SHAP and Kernel LIME are sampling-based approximations. For complex models with hundreds of features, larger permutation budgets are required for precise convergence.
4. Attribution Drift vs Predictive Performance: Attribution drift reflects changes in model reasoning and input distributions, but does not always imply a drop in top-line predictive accuracy (e.g. benign covariate shifts).
5. Sample Size Recommendations: Reliable quantile estimation requires at least 20 samples per partition. For sample sizes below 10, sampling variance increases.

---

Future Work

• Multidimensional and Sliced Wasserstein Distance: Extending DriftAuditor to capture joint multi-feature attribution distribution shifts via Sliced Wasserstein Distance or Maximum Mean Discrepancy (MMD).
• Feature-Weighted Drift Aggregation: Supporting user-defined feature importance weighting in Mean PSI calculations.
• TreeSHAP Integration: Adding native TreeSHAP support for tree ensembles to accelerate exact attribution computation.
• Counterfactual and Example-Based Dimensions: Developing additional audit dimensions for counterfactual recourse stability and influence function fidelity.
• Streaming Telemetry Connectors: Providing direct exporters for OpenTelemetry and Prometheus monitoring stacks.

---

Credits

Author: Fdere AI, Explainability Auditor Team
Governance Framework Compliance: Aligned with US SR 11-7 and EU AI Act Article 13 Verification Frameworks.

---

License

xai-auditor is licensed under the Apache License, Version 2.0. See the LICENSE file for details.