zidstats — two-sample testing, ranking, and distributional diagnostics

ZID compares two samples through their within-sample similarity structure and returns three complementary outputs: a permutation-calibrated p-value for equality testing, a score for ranking departures under a fixed protocol, and diagnostic readouts that report separate location-sensitive and dispersion-sensitive component magnitudes and, when direction is identifiable, distinguish under- from over-dispersion. It is designed to remain responsive to distributional differences—including scale, shape, dependence, and matched-moment structure—that can be compressed or missed by a single moment-based distance.

Paper: Hao Chen (2026), What FID Hides: Detecting, Ranking, and Diagnosing Deviations in Generative Evaluation, arXiv:2608.24881.

Dissimilarity-native: ZID accepts a finite, symmetric, nonnegative pairwise dissimilarity with zero self-distance—no coordinates, metric, or triangle inequality required. The off-diagonal entries must include a positive value and must not all be constant. These format conditions do not guarantee a nondegenerate permutation-null covariance: highly symmetric geometries can still produce a singular or numerically ill-conditioned covariance, in which case the test raises an error rather than reporting an unstable statistic. Object-valued observations such as networks, trees, shapes, and distributions can therefore be compared once their pooled pairwise dissimilarity matrix is supplied to zid_test_from_dissimilarity. Point matrices with a metric are the convenience case.

The core package is application-agnostic; generative-model evaluation helpers are provided separately in zidstats.generative.

Install

pip install zidstats        # numpy, scipy only

For repeated ordinary-GET calculations, optional compiled acceleration is available separately:

pip install "zidstats[get-accel]"

This extra accelerates only the strictly ordered ordinary-GET MST sequence. ZID, GPK, RISE, and GET-discrete do not require it; ordinary GET remains available through the default SciPy implementation when the extra is absent.

Quick start — one call

import numpy as np
from zidstats import zid_test

rng = np.random.default_rng(20260718)
centers = np.zeros((3, 64))
centers[:, :2] = 1.5 * np.array([
    [1.0, 0.0], [-0.5, np.sqrt(3) / 2], [-0.5, -np.sqrt(3) / 2]
])

# Reference: three balanced feature-space modes.
x_mode = rng.choice(3, 300, p=[1/3, 1/3, 1/3])
# Comparison sample: the third mode has almost disappeared.
y_mode = rng.choice(3, 300, p=[0.475, 0.475, 0.05])
X = centers[x_mode] + 0.7 * rng.standard_normal((300, 64))
Y = centers[y_mode] + 0.7 * rng.standard_normal((300, 64))

res = zid_test(X, Y, seed=17)
print(res)
# ZID: score=25.68 | p=0.002 | statistically distinguishable (alpha=0.05).
# Significant distribution difference. Component magnitudes: W=26.37;
# D=1.01, direction not assigned (D-component p=0.314).
# These are sensitivity readouts, not a causal decomposition.
res.pvalue, res.statistic, res.reject, res.verdict
res.statistic, res.extra["score"]           # same protocol-dependent ranking score
res.extra["W_component_magnitude"]          # nonnegative S_W magnitude
res.extra["D_component_magnitude"]          # nonnegative S_D magnitude
res.extra["W_component_pvalue"]             # component-level permutation tail p_W
res.extra["D_component_pvalue"]             # p_D used by the diagnostic gate
res.extra["D_ZID"]                           # gated signed diagnostic, or None
res.extra["dispersion_net_score"]           # continuous signed visualization score
res.extra["dispersion_diagnosis"]           # formal diagnosis label
res.extra["arms"]                           # signed member-level Z_W and Z_D values

zid_test runs the complete ZID procedure on two sample matrices and returns its permutation-calibrated p-value, ranking score, and diagnostic readouts in one result.

The public zid_score(X, Y) and zid_arms(X, Y) utilities return only the observed score or six observed member coordinates. They do not run the outer label permutations and therefore do not return a calibrated p-value or test decision.

The ZID extra schema uses component names only for aggregated quantities; member-level coordinates are contained exclusively in extra["arms"]. Protocol metadata are recorded alongside the readouts, including calibration="score_permutation", n_perm, bandwidths, and the RISE graph settings.

Main function inputs

zid_test(
    X,
    Y,
    alpha=0.05,
    n_perm=499,
    seed=None,
    metric="euclidean",
    **kwargs,
)

input	meaning	default
X	Real-valued reference sample matrix with shape (m, d); rows are finite observations and m >= 4.	required
Y	Real-valued comparison sample matrix with shape (n, d); it must have the same number of columns as X, contain only finite values, and have n >= 4.	required
alpha	Nominal level for the equality decision; the same numerical threshold is used by the prespecified dispersion-diagnosis gate.	0.05
n_perm	Number of label permutations used to calibrate the complete six-arm ZID score.	499
seed	None or a nonnegative integer seed for reproducible label permutations; booleans are rejected.	None
metric	Pairwise dissimilarity: a SciPy cdist metric name or a callable f(u, v) -> float. Callable metadata store module.qualified_name; parameters captured by partials, closures, or callable-object state must be recorded separately with the evaluation protocol.	"euclidean"
small_bandwidth_ratio	GPK-small bandwidth as a multiple of the GPK-med bandwidth.	0.175
sigma	Optional positive absolute bandwidth for GPK-med; if omitted, ZID uses the median positive pooled pairwise distance.	pooled median
k	Optional RISE neighborhood parameter, with N = m + n.	floor(N**0.65)
rank_type	RISE graph construction: "RgNN" or "RoNN".	"RgNN"

The final four entries are optional expert settings passed through **kwargs. For ordinary use, supply X and Y and leave the construction defaults fixed. With B = n_perm, the smallest attainable Monte Carlo p-value is 1 / (B + 1). Choose B so that this value is no larger than the intended significance level; for example, testing at alpha=0.001 requires at least n_perm=999.

Computational scope

ZID forms dense pooled dissimilarity, rank, and kernel matrices, so memory use is quadratic in N = m + n. With B = n_perm, the current outer-permutation implementation performs approximately O(B * N**2) work and therefore scales roughly linearly with B. For pooled samples in the several-thousand range, assess both memory and runtime on the intended hardware before committing to a large permutation budget.

Input forms: vectors, custom dissimilarities, or object data

For vector features, Euclidean dissimilarity is the default, but any SciPy cdist metric name can be substituted without changing the ZID construction:

res = zid_test(X, Y, metric="cosine")

An advanced callable metric receives two numeric rows u and v, each with shape (d,), and must produce finite, symmetric, nonnegative pairwise dissimilarities with zero self-distance. It is for custom numeric-vector geometry, not for passing arbitrary Python objects directly; the resulting pooled matrix is validated before the test runs.

Networks, trees, shapes, distributions, and other object-valued observations do not need vector coordinates. Compute a pooled pairwise dissimilarity matrix D, place the reference objects first, and run ZID directly:

from zidstats import zid_test_from_dissimilarity

# D[i, j] may be network distance, tree distance, shape distance, or another
# domain-specific dissimilarity; it need not be Euclidean or satisfy the triangle inequality.
res = zid_test_from_dissimilarity(D, n_x=len(reference_objects))

The same route supports complex-valued observations: first compute an application-appropriate real-valued dissimilarity matrix. Raw complex sample matrices are rejected rather than silently discarding their imaginary parts.

zid_test_from_dissimilarity inputs

zid_test_from_dissimilarity(
    D,
    n_x,
    alpha=0.05,
    n_perm=499,
    seed=None,
    **kwargs,
)

input	meaning	default
D	Real-valued pooled pairwise dissimilarity matrix with shape (N, N). It must be finite, symmetric, nonnegative, have a zero diagonal, contain a positive off-diagonal value, and not have a constant off-diagonal. Rows and columns [:n_x] contain the reference sample; [n_x:] contain the comparison sample. A matrix satisfying these format conditions can still yield a singular permutation covariance, which is reported as an error.	required
n_x	Number of reference observations and split point in D; the split must leave at least four observations in each sample.	required
alpha	Nominal level for the equality decision; the same numerical threshold is used by the prespecified dispersion-diagnosis gate.	0.05
n_perm	Number of label permutations used to calibrate the complete six-arm ZID score.	499
seed	None or a nonnegative integer seed for reproducible label permutations; booleans are rejected.	None
small_bandwidth_ratio	GPK-small bandwidth as a multiple of the GPK-med bandwidth.	0.175
sigma	Optional positive absolute bandwidth for GPK-med; if omitted, ZID uses the median positive off-diagonal entry of D.	pooled median
k	Optional RISE neighborhood parameter.	floor(N**0.65)
rank_type	RISE graph construction: "RgNN" or "RoNN".	"RgNN"

The final four entries are optional expert settings passed through **kwargs. Unlike zid_test, this function has no metric argument because the pooled dissimilarities have already been supplied in D.

No multidimensional scaling or Euclidean embedding is performed.

All four public ZID test paths apply the same duplicate check exactly once. DuplicateSampleWarning identifies near-zero cross-sample dissimilarities that suggest near-matching observations; repeated positive distance values alone do not trigger it. For an otherwise valid ZID input, a zero cross-sample dissimilarity always triggers the warning. For nonzero near-matches, the heuristic suppresses the warning when comparable within-sample near-repetition is pervasive.

What the ZID score/test does. ZID takes the signed Z_W/Z_D arms from RISE, GPK at the median bandwidth, and GPK at a small (0.175×median by default) bandwidth, maps their magnitudes through the conventional two-sided normal-reference tail 2 * Phi_bar(abs(Z)), and flat-Simes-combines all six into an unbounded ranking score that is coordinatewise monotone in the observed arm magnitudes. Larger values indicate a more extreme standardized departure from the reference when sample size, representation, reference, and preprocessing are held fixed. The returned p-value is the plus-one label-permutation tail of that same score, so it preserves dependence among the arms and is finite-sample valid under exchangeability. This validity requires the representation, pooled dissimilarity, and tuning choices to be fixed independently of the two-sample labels. If any of them are selected using those labels, the complete selection procedure must be repeated within every relabeling.

Reference orientation matters only for the signed diagnostics. Swapping X and Y leaves the ZID score and its permutation reference distribution unchanged, while reversing the member-level Z_D arms, dispersion_net_score, and any supported D_ZID readout; it also interchanges supported under-dispersion and over-dispersion diagnoses. The reported Monte Carlo p-value is subject to ordinary Monte Carlo variation from the sampled relabelings. Put the reference first for a directional read.

How to read the diagnostics. W_component_magnitude and D_component_magnitude are the nonnegative aggregated W- and D-component magnitudes, denoted S_W and S_D, not additive parts of the score or a causal decomposition. D_component_magnitude is always nonnegative, whereas dispersion_net_score applies the sign of the sum of the three member-level Z_D arms and is available continuously for visualization. With X designated as the reference, the coherence-qualified D_ZID equals -D_component_magnitude for under-dispersion and +D_component_magnitude for over-dispersion; it is undefined for every other diagnosis and is returned as Python None. These labels have a dispersion interpretation only when that interpretation is supported by the chosen dissimilarity and application. Let p_W = res.extra["W_component_pvalue"] and p_D = res.extra["D_component_pvalue"] denote the permutation tails of the aggregated three-arm W- and D-component magnitudes. These are component-level diagnostic summaries; the equality decision uses the permutation p-value of the complete six-arm ZID score. In particular, p_D is a prespecified diagnostic gate rather than an additional, separately error-controlled test. Only p_D is used by the formal dispersion_diagnosis, which requires p_D <= alpha. A member j is reference-active when its normal-reference tail 2 * Phi_bar(abs(Z_D^(j))) <= alpha. Both this gate and the reference-active member threshold use the alpha supplied to zid_test. If p_D > alpha, the diagnosis is not assigned; if p_D <= alpha but no member is reference-active, it is ambiguous; and conflicting reference-active member signs are reported as member-sign conflict. All member arms remain available for inspection; D_reference_active_members records the members in the descriptive reference-active set.

For a quick single-member check without permutation calibration, use method="rise": RISE uses the analytic asymptotic χ²₂ reference for Z_W²+Z_D²; it is deterministic and requires no permutation. RISE still constructs dense N × N dissimilarity and rank matrices, so its memory use remains quadratic in the pooled sample size.

RISE tied-distance extension. The RISE paper and the original R implementation in GraphRankTest do not define the label-invariant averaging rule used here for equal-distance neighbors. zidstats adds the following convention. For RgNN, tied neighbors receive the mean position weight over the rank positions occupied by their tie block. For RoNN, if a tie block crosses the k-nearest-neighbor cutoff, each tied observation receives its inclusion probability under uniform tie-breaking. For each unordered pair, the two directional neighbor-inclusion probabilities are combined into the probability that at least one observation selects the other; this probability weights the edge's global similarity midrank. These extensions change nothing when neighbor distances are distinct and remove dependence on pooled-row order when ties occur. When equal distances actually alter the RISE weights, analytic standalone RISE emits RiseTieWarning: the published asymptotic reference has not been established specifically for this project-authored extension. Use calibration="permutation" for a label-permutation calibration of that same tie-averaged statistic. ZID uses its outer score permutation with the same label-invariant weights.

Routing — which member when

Use ZID when broad coverage is desired and the departure type is not prespecified. If the application instead provides a specific geometric, computational, or discrete-data reason to use a standalone specialist, route by geometry rather than by the smallest p-value:

your situation	route →	why
broad coverage / departure type not prespecified	ZID	one six-arm ranking score, its calibrated permutation tail, and conditional diagnostic readouts
quick check / avoid permutation calibration	RISE (method="rise")	single-member test using an analytic asymptotic χ²₂ reference; deterministic after the pairwise dissimilarities are computed
non-Euclidean / manifold / graph features	RISE	local rank graph; a global kernel bandwidth can smooth away dispersion differences on a manifold
plain Euclidean / Gaussian, prioritizing location/dispersion sensitivity	GPK	median-bandwidth kernel test with strong sensitivity to high-dimensional location and dispersion alternatives
need member-level inspection	ZID (inspect arms)	retains all six signed arms and reports whether the reference-active dispersion signs are coherent
repeated observations or duplicated rows	GET-discrete, GPK, RISE, or ZID	GET-discrete implements the repeated-observation graph extension; GPK handles ties directly; standalone RISE and the RISE member inside ZID use label-invariant tie handling
other tied pairwise dissimilarities	GPK, RISE, or ZID	GET-discrete is designed for duplicated-row equivalence classes; GPK handles ties directly, and RISE uses label-invariant tie handling
need component-level diagnosis	ZID components	read W_component_magnitude, D_component_magnitude, and dispersion_diagnosis; inspect extra["arms"] for member-level attribution

Methods

The advanced two_sample_test(..., method=...) dispatcher exposes RISE, GPK, fGPK, GET, and GET-discrete as standalone tests. ZID combines RISE with median- and small-bandwidth GPK; neither GET method is a ZID arm. The dispatcher is not a second ZID entry point in the public workflow. Every package-level test entry point returns the same TestResult object. Formula-level implementations used for verification remain in the method submodules and are not part of the package-level workflow. For a pooled dissimilarity matrix, the matching router is two_sample_test_from_dissimilarity.

from zidstats import two_sample_test

rise = two_sample_test(X, Y, method="rise")
gpk_med = two_sample_test(X, Y, method="gpk", n_perm=499, seed=0)
gpk_small = two_sample_test(
    X, Y, method="gpk", bandwidth_ratio=0.175, n_perm=499, seed=0
)
fgpk = two_sample_test(X, Y, method="fgpk")
get = two_sample_test(X, Y, method="get")
get_discrete = two_sample_test(
    X, Y, method="get-discrete", approach="averaging"
)

# RISE, GET, and GET-discrete can instead use label-permutation calibration.
rise_perm = two_sample_test(
    X,
    Y,
    method="rise",
    calibration="permutation",
    n_perm=499,
    seed=0,
)

RISE, GET, and GET-discrete use their published analytic asymptotic chi-square references by default. For any of these three methods, set calibration="permutation" to calibrate the same quadratic statistic by label relabeling. The selected calibration determines TestResult.pvalue; under permutation calibration, extra also retains pvalue_asymptotic and the asymptotic component tails. Because n_perm is unused by analytic RISE, GET, GET-discrete, and fGPK, those calls may set n_perm=0; ZID, GPK, and explicitly permutation-calibrated graph tests require a positive value.

For standalone GPK, omitting both bandwidth arguments uses the pooled median bandwidth. Set bandwidth_ratio=0.175 for 0.175 * sigma_med, or pass an absolute sigma; the two arguments are mutually exclusive. small_bandwidth_ratio configures the fine-scale member inside ZID and does not alter standalone GPK.

method="get-discrete" implements the repeated-observation graph extension of Zhang and Chen (2022). It first forms a graph on the K distinct pooled values, using the union of all optimal MST edges when the graph is not unique, and then removes observation-level tie-breaking in one of two published ways. The default approach="averaging" averages the within-sample edge counts over all induced optimal graphs; approach="union" uses their observation-level union. Ordinary method="get" continues to use edge-disjoint MSTs and raises an error only when that requested MST sequence is actually non-unique. It never selects an arbitrary optimal tree. Set k explicitly to use the denser graph formed from the first k nearest-neighbor-link (NNL) graphs; the GET-discrete default is k=1.

fGPK is a separate fast analytic test, not an analytic calibration of the GPK quadratic statistic. Its reported statistic is the minimum of the one-sided normal-reference tails for Z_W_r1 and Z_W_r2 and the two-sided tail for Z_D; its p-value applies the published three-way Bonferroni adjustment. The published fGPK specification uses the ordered pair of weights 1.2 and 0.8; consequently, swapping the two samples can slightly change its statistic and may change decisions near the significance boundary.

method	type	p-value	notable strengths	tuning (default)
zid	six-arm score/test (default)	Monte Carlo permutation tail of the ZID score	broad coverage across controlled departures	sigma=median, small=0.175×; n_perm=499
gpk	kernel	Monte Carlo permutation tail of the GPK statistic	location + dispersion (Euclidean)	bandwidth_ratio=1 (median)
fgpk	fast kernel test	analytic asymptotic three-way Bonferroni test	location + dispersion	bandwidth_ratio=1 (median)
rise	graph-induced rank	analytic asymptotic χ²₂ (default) or Monte Carlo permutation	shape + multimodality + dependence	k = ⌊N^0.65⌋
get	unique-MST edge-count	analytic asymptotic χ²₂ (default) or Monte Carlo permutation	scale + dependence	k = ⌊N^0.5⌋
get-discrete	repeated-observation edge-count	analytic asymptotic χ²₂ (default) or Monte Carlo permutation	discrete or repeated data	averaging over observation-level optimal graphs; one NNL graph on K distinct values

The family tests are nonparametric implementations of the published mathematical definitions. ZID and GPK use label permutations; RISE, GET, and GET-discrete default to their published analytic asymptotic nulls and also offer permutation calibration. See PROVENANCE.md for the implementation and licensing record.

Attribution arms. For RISE, GET, and GET-discrete, Z_W and Z_D are asymptotically N(0,1) and asymptotically independent under their published null conditions, giving the χ²₂ reference for Z_W²+Z_D². They provide two-coordinate attribution: Z_W records aligned within-sample displacement and is often location-sensitive; Z_D records opposing displacement and is dispersion-sensitive in validated scale regimes. Under the default analytic calibration, each has an asymptotic p-value (2·Φ(−|·|)); under permutation calibration, Z_W_pvalue and Z_D_pvalue are two-sided Monte Carlo tails. For GPK, the equal-weight Z_W coordinate exposed as extra["Z_W"] is the standardized MMD-equivalent coordinate and need not be N(0,1); its Monte Carlo permutation p-value is returned as extra["Z_W_pvalue"]. The GPK Z_D coordinate is asymptotically N(0,1); its Monte Carlo component tail is returned as extra["Z_D_pvalue"] for optional inspection. The default GPK verdict follows only the permutation p-value of the GPK quadratic statistic. The separate fGPK test instead exposes the weighted, asymptotically N(0,1) coordinates Z_W_r1 and Z_W_r2 together with Z_D.

Note on "kernel" (≠ MMD)

GPK uses a kernel representation but is not the ordinary scalar MMD test. For any m,n >= 2 and a fixed pooled kernel matrix, the unbiased empirical squared MMD is a positive multiple of the centered W statistic, so its permutation-standardized statistic is the Z_W coordinate. GPK also retains the complementary signed Z_D coordinate. Under high-dimensional dispersion alternatives, opposing within-sample shifts can cancel in Z_W while reinforcing in Z_D.

Funding

This material is based upon work supported by the National Science Foundation under Award No. DMS-2311399. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Cite

ZID: Hao Chen (2026), What FID Hides: Detecting, Ranking, and Diagnosing Deviations in Generative Evaluation, arXiv:2608.24881.

Constituent methods: GPK (Song & Chen 2024, Biometrika 111(3):755–770), RISE (Zhou & Chen 2023, COLT/PMLR 195), GET (Chen & Friedman 2017, JASA 112(517):397–409), and the repeated-observation graph extension (Zhang & Chen 2022, Statistica Sinica 32:391–415).