MultiTest — Global Tests for Multiple Hypothesis Testing
MultiTest provides several methods for combining p-values with a focus on detecting rare and weak effects.
Higher Criticism variants
Each HC variant is exposed as its own method so that the standardization choice is explicit rather than a constructor argument.
| Method | Standardization | P-value range considered |
|---|---|---|
MultiTest.hc |
Beta-distribution std (default, alias for hc_star) | (1/n, γ] |
MultiTest.hc_dj2004 |
Donoho-Jin 2004 [1] – observed p-value std | (0, γ] |
MultiTest.hc_dj2008 |
Donoho-Jin 2008 [2] – theoretical uniform std | (0, γ] |
MultiTest.hc_beta |
Beta-distribution std | (0, γ] |
MultiTest.hc_star |
Beta-distribution std | (1/n, γ] (HCdagger [1]) |
Every HC method accepts:
gamma– upper fraction of sorted p-values to consider. Only the p-values ranked in positions 1 through ⌊γ·n⌋ (i.e. the smallest γ fraction) enter the HC statistic. Defaults to'auto'(see below).return_threshold– ifTrue, returns(hc_score, threshold_pval); otherwise returns just the score (defaultFalse).
Default gamma and why it matters
When gamma='auto' the upper limit is set to
γ = log(n) / sqrt(n)
This keeps HC focused on the regime where it has the most power. Signals denser than roughly 1/√n features are detectable by simpler methods (e.g. the average z-score), so extending HC beyond γ = log(n)/√n adds noise without gaining power. The log(n) factor provides a small safety margin above the 1/√n threshold.
Other methods
MultiTest.berkjones/berkjones_plus– Berk-Jones statistic [3]MultiTest.fdr– False-discovery rate functionalMultiTest.fdr_control– Benjamini-Hochberg FDR controlMultiTest.bonferroni/neg_log_minp– Bonferroni-style inferenceMultiTest.fisher– Fisher's method to combine p-values
In all cases, reject the null for large values of the test statistic.
Quick start
import numpy as np
from scipy.stats import norm
from multitest import MultiTest
n = 100
z = np.random.randn(n)
pvals = 2 * norm.cdf(-np.abs(z))
mt = MultiTest(pvals)
# Default HC score
hc = mt.hc()
# HC score + threshold p-value
hc, hct = mt.hc(return_threshold=True)
# Berk-Jones
bj = mt.berkjones()
ii = np.arange(n)
print(f"HC = {hc:.3f}, features below HCT: {ii[pvals <= hct]}")
print(f"Berk-Jones = {bj:.3f}")
Choosing an HC variant
hcis the recommended starting point. It is an alias forhc_star(beta-distribution standardization, p-values restricted to the range (1/n, γ]).hc_starignores p-values below 1/n (sample-size adjusted, HCdagger [1]), using beta-distribution standardization.hc_betauses beta-distribution standardization over the full range (0, γ].hc_dj2008uses the expected (theoretical) uniform mean and std for normalization, matching the formulation in [2].hc_dj2004uses the observed p-value standard deviation as denominator. This makes it more sensitive to extreme p-values but also increases variance under the null.
Use cases
This package was used to obtain evaluations reported in [4] and [5].
References
[1] Donoho, David L. and Jin, Jiashun. "Higher criticism for detecting sparse heterogeneous mixtures." The Annals of Statistics 32, no. 3 (2004): 962-994.
[2] Donoho, David L. and Jin, Jiashun. "Higher criticism thresholding: Optimal feature selection when useful features are rare and weak." Proceedings of the National Academy of Sciences, 2008.
[3] Moscovich, Amit, Boaz Nadler, and Clifford Spiegelman. "On the exact Berk-Jones statistics and their p-value calculation." Electronic Journal of Statistics 10 (2016): 2329-2354.
[4] Donoho, David L. and Alon Kipnis. "Higher criticism to compare two large frequency tables, with sensitivity to possible rare and weak differences." The Annals of Statistics 50, no. 3 (2022): 1447-1472.
[5] Kipnis, Alon. "Unification of rare/weak detection models using moderate deviations analysis and log-chisquared p-values." Statistica Sinica, 2025.
Metadata
Release files for multiple-hypothesis-testing 0.1.16
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| multiple_hypothesis_testing-0.1.16.tar.gz | 7.4 kB | Without provenance | Details for multiple_hypothesis_testing-0.1.16.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| multiple_hypothesis_testing-0.1.16-py3-none-any.whl | Python 3 | none | any | Details for multiple_hypothesis_testing-0.1.16-py3-none-any.whl |
Total release size: 15.6 kB
Release files / multiple_hypothesis_testing-0.1.16.tar.gz
| Download URL | multiple_hypothesis_testing-0.1.16.tar.gz |
|---|---|
| Size | 7.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5e5f60f8cd5a24514f900c3b37350f52046fc222383212500c3bdb3f0c7615ee
|
|
BLAKE2b-256 checksum How to use checksums |
ae535d86d3f2f4413f6ad2285669b96d6f54b5fcd561e1af956132c5b7a876b8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.6
|
Release files / multiple_hypothesis_testing-0.1.16-py3-none-any.whl
| Download URL | multiple_hypothesis_testing-0.1.16-py3-none-any.whl |
|---|---|
| Size | 8.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1b956d0233f72b02968fce4c1da871462998f62dc31ddf946358e8546dd12ff6
|
|
BLAKE2b-256 checksum How to use checksums |
228fbe3ff47ceb7af6d573a432dc28ae23be31fc2ccbfa5608b01958196475d9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.6
|
Security for multiple-hypothesis-testing 0.1.16
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.