ancestryaudit 0.3.5

Bias detection and correction framework for genomic cancer AI

AncestryAudit

Bias detection and correction framework for genomic cancer AI.

Detects ancestry-linked performance gaps in copy number variation (CNV)-based cancer classifiers, applies McNemar's paired test for correction validation, and generates structured audit reports.

Developed from research on ancestry bias in TCGA-LIHC/STAD/ESCA classification (Yergaliyeva, 2026).

v0.3.0 — Statistical overhaul: Original bootstrap t-test replaced with label-permutation test (FPR: 85% → 0%). Correction module rewritten around McNemar's paired test. Power analysis added. Full methodology in paper.

---

Installation

pip install ancestryaudit
# or from source:
pip install -e .

---

Input Format

AncestryAudit works on any CNV feature matrix:

Format	Shape	Notes
np.ndarray	(n_samples, n_genes)	Continuous copy-number values
pd.DataFrame	(n_samples, n_genes)	Column names = gene identifiers

Column values: continuous copy number (e.g. TCGA ABSOLUTE pipeline output, where 2.0 = normal diploid, >2 = amplification, <2 = deletion).

Labels: binary integer (0 or 1), one per sample.

Models: any scikit-learn compatible estimator with fit / predict interface.

---

Quick Start

from ancestryaudit import AncestryAuditFramework
from sklearn.linear_model import LogisticRegression

framework = AncestryAuditFramework()

# Step 0: Check if you have enough data (run this first)
power = framework.power_analysis(n_source=451, n_target=242, expected_gap_pp=3.0)
# → UNDERPOWERED: need n_target≈836 for 80% power at 3pp

# Step 1: Detect ancestry-linked performance gap
report = framework.audit(
    LogisticRegression(max_iter=1000),
    X_western, y_western,
    X_asian,   y_asian
)
print(f"Gap: {report.gap_pp:.2f}pp, p={report.p_value:.4f}")
print(f"Recommendation: {report.recommendation}")

Output:

Gap: +2.39pp, p=0.0069
Recommendation: correction_required

---

Full Pipeline

from ancestryaudit import AncestryAuditFramework
from sklearn.linear_model import LogisticRegression

framework = AncestryAuditFramework(
    random_state=42,
    threshold_pp=2.0,
    threshold_p=0.05
)

# ── Step 0: Power analysis (before anything else) ──────────────────────────
power = framework.power_analysis(
    n_source=451, n_target=242, expected_gap_pp=3.0
)
# Tells you whether your data is sufficient before you run the audit

# ── Step 1: Filter population-stratification noise (optional) ──────────────
X_western_filtered, kept_genes, filter_log = framework.filter_stratification_noise(
    X_western_df, gene_list
)

# ── Step 2: Audit ──────────────────────────────────────────────────────────
audit_report = framework.audit(
    LogisticRegression(max_iter=1000),
    X_western_filtered, y_western,
    X_asian_filtered,   y_asian
)
print(audit_report)
# AuditReport(gap=+2.39pp, p=0.0069, d=1.52,
#             null_CI=[-2.10, 2.15], recommendation='correction_required')

# ── Step 3: Correct ────────────────────────────────────────────────────────
if audit_report.recommendation == "correction_required":
    corrected_model, correction_report = framework.correct(
        LogisticRegression(max_iter=1000),
        X_western_filtered, y_western,
        X_asian_labeled,    y_asian_labeled,
        n_samples=75
    )
    print(correction_report)
    # CorrectionReport(delta=+1.43pp, McNemar p=0.031,
    #                  direction='fine-tuned better')

# ── Step 4: Validate ───────────────────────────────────────────────────────
validation_report = framework.validate(
    corrected_model,
    X_asian_holdout, y_asian_holdout
)

# ── Step 5: Report ─────────────────────────────────────────────────────────
framework.generate_report("my_audit_report.json")
framework.summary()

---

API Reference

AncestryAuditFramework

Method	Description	Returns
power_analysis(n_source, n_target, expected_gap_pp)	Check data sufficiency before audit	dict
audit(model, X_source, y_source, X_target, y_target)	Detect gap (permutation test)	AuditReport
filter_stratification_noise(X, gene_list)	Remove OR/pseudogene columns	(X_filtered, kept_genes, filter_log)
correct(model, X_source, y_source, X_target_labeled, y_target_labeled, n_samples)	Fine-tune + McNemar test	(corrected_model, CorrectionReport)
validate(corrected_model, X_holdout, y_holdout)	Post-correction audit	ValidationReport
generate_report(save_path)	Full JSON report	dict
summary()	Print pipeline summary	str

AuditReport fields

Field	Type	Description
gap_pp	float	Accuracy gap in percentage points (positive = source better)
p_value	float	Two-sided p-value from label-permutation test
cohen_d	float	Between-group effect size (pooled SD of per-sample correctness)
null_ci	tuple	2.5/97.5 percentiles of permutation null — not a CI on the gap
source_accuracy	float	Model accuracy on held-out source data
target_accuracy	float	Model accuracy on target data
n_source	int	Source sample count
n_target	int	Target sample count
recommendation	str	"correction_required" or "no_action"

CorrectionReport fields

Field	Type	Description
delta_pp	float	Accuracy improvement on holdout (pp)
p_value	float	McNemar's test p-value
n_used	int	Target samples used for fine-tuning
n_holdout	int	Holdout samples used for evaluation
mcnemar	dict	b, c, p_value, direction, test_used
refit_robustness	dict	5 refits varying model random_state (split fixed)
direction_confirmed	bool	True if McNemar p<0.05 and fine-tuned better
baseline_accuracy	float	Source-only accuracy on full target
corrected_accuracy	float	Fine-tuned accuracy on full target

ValidationReport fields

Field	Type	Description
pre_gap	float	Performance gap before correction (pp)
post_gap	float	Performance gap after correction (pp)
correction_magnitude	float	pre_gap - post_gap
improvement_pp	float	Accuracy improvement on target (pp)

---

Statistical Design

audit() — Permutation test

Why not bootstrap? The original implementation used a bootstrap t-test that produced 85% false positive rate on null data (identical distributions). Root cause: testing whether a point estimate's resampling variance excludes zero is circular — it will always reject H0 on finite data regardless of whether a true effect exists.

Correct approach: Label-permutation test. Shuffle which samples are "source test" vs "target" 1000 times, recompute the gap under each shuffle, locate the observed gap in that null distribution. Verified: 0% FPR across 800 null trials.

correct() — McNemar's paired test

Why not bootstrap on folds? Three attempts at fold-based bootstrap (independent seeds, StratifiedKFold, RepeatedStratifiedKFold) all produced inflated FPR (22–26%) due to holdout overlap between folds at n_target=242.

Correct approach: McNemar's exact/chi-square test on a single fixed holdout. Tests whether the two classifiers disagree asymmetrically — exactly the right question. Verified: 0% FPR across 500 null trials.

---

Power Analysis

Critical: run power_analysis() before audit(). At TCGA-scale Asian cohorts (n≈242), the minimum detectable effect is ~8–11pp. Realistic ancestry-linked gaps are 1–5pp. The test is structurally underpowered for this problem at current sample sizes.

result = framework.power_analysis(
    n_source=451, n_target=242, expected_gap_pp=3.0
)
# → UNDERPOWERED
# → n_target needed: ≈836 (both sides must scale together)
# → Minimum viable study: n_target≥750, n_source≥1,500

Reference: Saldanha et al. (2024, Nature Medicine) documented 3–16pp ancestry-linked gaps in TCGA-trained cancer AI. Our MDE of 8.47pp exceeds the upper end of this range.

---

Filtering Details

filter_stratification_noise removes three gene categories:

• Olfactory receptor genes (OR*) — ancestry-linked CNV unrelated to cancer
• Pseudogenes (*P, *P1, *P2, …) — non-functional, high stratification signal
• Uncharacterized loci (names containing .) — no biological interpretation

Methods disclosure: Feature space defined using all samples prior to train/test split (bounded data snooping). No label information used. (Kaufman et al., 2012)

---

Expected Results

Disclaimer: Results depend on model, train/test split, and preprocessing. Directional consistency (not numerical identity) with paper results is the correct validation criterion.

Null calibration (verified):

• audit() permutation test: ~1-2% FPR (independent 800-trial replication)
• correct() McNemar test: 1.7% FPR (independent 300-trial replication)
• Both well below 5% nominal α; McNemar exact is conservative by construction

Paper reproduction:

Metric	Paper	Library
Mean PGI (LIHC/STAD)	+2.39pp	+2.393pp
Algorithms positive	7/7	7/7
p-value (permutation)	0.0048	verified

---

Tests

python tests/test_null_calibration.py   # FPR ≤ 20% on null data
python tests/test_power_calibration.py  # flip_rate calibration regression

---

Citation

Yergaliyeva, D. (2026). Statistical pitfalls in ancestry-stratified
cancer genomic AI: Power analysis, circular-inference correction,
and structural confounds in TCGA CNV data.
[Manuscript in preparation]

---

License

MIT License. Copyright (c) 2026 Dana Yergaliyeva.