Skip to main content

zedstat

https://zed.uchicago.edu/logo/logo_zedstat.png https://zenodo.org/badge/529991779.svg
Author:

ZeD@UChicago <zed.uchicago.edu>

Description:

Tools for ML statistics

Documentation:

https://zeroknowledgediscovery.github.io/zedstat/

Example:

https://github.com/zeroknowledgediscovery/zedstat/blob/master/examples/example1.ipynb

Additional usage examples

1. Export operating characteristics at a chosen prevalence

import pandas as pd
from zedstat import zedstat

roc_df = pd.read_csv("roc.csv")

zt = zedstat.processRoc(
    df=roc_df,
    order=3,
    total_samples=100000,
    positive_samples=100,
    alpha=0.01,
    prevalence=0.002,
)

zt.smooth(STEP=0.001)
zt.allmeasures(interpolate=True)
zt.usample(precision=3)
zt.getBounds()

out = zt.get().join(zt.df_lim["U"], rsuffix="_upper").join(zt.df_lim["L"], rsuffix="_lower")
out.to_csv("roc_operating_characteristics.csv")

2. Retrieve threshold-level PPV from a score

This uses the ROC-derived operating characteristics and prevalence to estimate the positive predictive value associated with using a score as a decision threshold.

example_scores = [0.10, 0.20, 0.30]

ppv_at_threshold = zt.score_to_threshold_ppv(
    example_scores,
    regen=True,
    STEP=0.001,
    precision=3,
    interpolate=True,
    convexify=False,
)

threshold_ppv_df = pd.DataFrame({
    "score": example_scores,
    "threshold_ppv": ppv_at_threshold,
})

display(threshold_ppv_df)

3. Held-out calibration with isotonic regression

The calibration module is separate from the ROC processing utilities and is imported as:

from zedstat import calibration

res = calibration.heldout_isotonic_calibration_with_bootstrap(
    df,
    score_col="predicted_risk",
    label_col="target",
    test_size=0.25,
    random_state=4,
    lower_score_is_risk=False,
    target_prevalence=None,
    n_bins=100,
    n_boot=1000,
    calibration_df_path="calibration_df_SISA.csv",
    plot="calibration_SISA.pdf",
)

print(res["summary"])
display(res["calibration_table"])

4. Convert raw scores to calibrated probabilities

The held-out calibration routine returns the fitted isotonic regression model in res["iso_model"]. This can be applied to any score vector.

import numpy as np
import pandas as pd

example_scores = [0.10, 0.20, 0.30]
example_scores_arr = np.asarray(example_scores, dtype=float)

calibrated_probs = res["iso_model"].predict(example_scores_arr)

calibrated_df = pd.DataFrame({
    "score": example_scores,
    "calibrated_probability": np.asarray(calibrated_probs, dtype=float),
})

display(calibrated_df)

5. Format calibration summary for a manuscript table

def format_calibration_summary_df(summary):
    import numpy as np
    import pandas as pd

    summary = pd.Series(summary)

    ci_map = {
        "auc_raw_test": ("auc_raw_ci_low", "auc_raw_ci_high"),
        "auc_calibrated_test": ("auc_calibrated_ci_low", "auc_calibrated_ci_high"),
        "brier_raw_test": ("brier_raw_ci_low", "brier_raw_ci_high"),
        "brier_calibrated_test": ("brier_calibrated_ci_low", "brier_calibrated_ci_high"),
        "calibration_intercept_test": ("calibration_intercept_ci_low", "calibration_intercept_ci_high"),
        "calibration_slope_test": ("calibration_slope_ci_low", "calibration_slope_ci_high"),
    }

    skip_keys = {
        "auc_raw_ci_low", "auc_raw_ci_high",
        "auc_calibrated_ci_low", "auc_calibrated_ci_high",
        "brier_raw_ci_low", "brier_raw_ci_high",
        "brier_calibrated_ci_low", "brier_calibrated_ci_high",
        "calibration_intercept_ci_low", "calibration_intercept_ci_high",
        "calibration_slope_ci_low", "calibration_slope_ci_high",
    }

    rows = []
    for key, val in summary.items():
        if key in skip_keys:
            continue

        value_str = "" if pd.isna(val) else f"{float(val):.3f}"

        if key in ci_map:
            lo_key, hi_key = ci_map[key]
            lo = summary.get(lo_key, np.nan)
            hi = summary.get(hi_key, np.nan)
            if pd.notna(lo) and pd.notna(hi):
                value_str = f"{float(val):.3f} ({float(lo):.3f}, {float(hi):.3f})"

        rows.append({"variable": str(key), "value": value_str})

    return pd.DataFrame(rows, columns=["variable", "value"])

summary_df = format_calibration_summary_df(res["summary"])
display(summary_df)

Feature explanations

processRoc

processRoc is the main ROC post-processing class. It takes an empirical ROC curve with false positive rate and true positive rate, augments it with operating metrics, and allows interpolation, confidence bounds, and interpretation at chosen operating points.

smooth(STEP, interpolate, convexify)

This function regularizes the empirical ROC curve. If convexify=True, the upper ROC hull is computed so that dominated operating points are removed. If interpolate=True, the curve is resampled on a uniform false positive rate grid.

Let the ROC curve be represented as points

\begin{equation*} \{(f_i, t_i)\}_{i=1}^m \end{equation*}

where \(f_i\) is the false positive rate and \(t_i\) is the true positive rate. After optional convexification and interpolation, these are resampled onto a uniform grid in \(f\).

allmeasures(prevalence)

This computes threshold-level operating measures from sensitivity, specificity, and prevalence.

Let

\begin{equation*} \mathrm{TPR}(t) = P(\hat Y_t = 1 \mid Y=1), \qquad \mathrm{FPR}(t) = P(\hat Y_t = 1 \mid Y=0), \end{equation*}

and let prevalence be

\begin{equation*} \pi = P(Y=1). \end{equation*}

Then:

\begin{equation*} \mathrm{Specificity}(t) = 1 - \mathrm{FPR}(t) \end{equation*}
\begin{equation*} \mathrm{PPV}(t) = \frac{\mathrm{TPR}(t)\pi} {\mathrm{TPR}(t)\pi + \mathrm{FPR}(t)(1-\pi)} \end{equation*}
\begin{equation*} \mathrm{NPV}(t) = \frac{(1-\mathrm{FPR}(t))(1-\pi)} {(1-\mathrm{FPR}(t))(1-\pi) + (1-\mathrm{TPR}(t))\pi} \end{equation*}
\begin{equation*} \mathrm{Accuracy}(t) = \pi \mathrm{TPR}(t) + (1-\pi)(1-\mathrm{FPR}(t)) \end{equation*}
\begin{equation*} \mathrm{LR}^+(t) = \frac{\mathrm{TPR}(t)}{\mathrm{FPR}(t)} \end{equation*}
\begin{equation*} \mathrm{LR}^-(t) = \frac{1-\mathrm{TPR}(t)}{1-\mathrm{FPR}(t)} \end{equation*}

These are threshold-level decision quantities. They describe the performance of classifying everyone whose score crosses the threshold.

usample(precision)

This resamples the metric tables on a uniform false positive rate grid, typically for downstream lookup and plotting. The grid spacing is controlled by the decimal precision.

getBounds(total_samples, positive_samples, alpha)

This computes pointwise confidence bounds for the operating measures using Wilson intervals for sensitivity and specificity, then propagates these to PPV, NPV, accuracy, and likelihood ratios.

If \(n_1\) is the number of positive cases and \(n_0\) is the number of negative cases, then Wilson intervals are first computed for

\begin{equation*} \mathrm{TPR}(t) \quad \text{using } n_1 \end{equation*}

and for

\begin{equation*} \mathrm{Specificity}(t) = 1-\mathrm{FPR}(t) \quad \text{using } n_0. \end{equation*}

The derived measures are then bounded by substituting lower and upper values of sensitivity and specificity into the formulas above.

auc()

The area under the ROC curve is

\begin{equation*} \mathrm{AUC} = \int_0^1 \mathrm{TPR}(f)\, df \end{equation*}

where \(f\) is false positive rate. Numerically this is computed by trapezoidal integration over the processed ROC curve. Confidence intervals can be estimated either analytically or by bootstrap when raw scores and labels are available.

operating_zone(LRplus, LRminus)

This identifies practically useful threshold regions subject to likelihood-ratio constraints, for example high precision or high sensitivity operating points. Internally, the method searches the set of thresholds satisfying

\begin{equation*} \mathrm{LR}^+(t) > c_1, \qquad \mathrm{LR}^-(t) < c_2 \end{equation*}

for user-specified constants \(c_1\) and \(c_2\).

interpret(fpr, number_of_positives, five_yr_survival, factor)

This converts operating characteristics into expected counts for an interpretable hypothetical population.

If \(P\) is the chosen number of true positives in the target population and prevalence is \(\pi\), then the implied number of negatives is

\begin{equation*} N = P \frac{1-\pi}{\pi}. \end{equation*}

Given sensitivity and PPV at the selected operating point, the method estimates true positives, false positives, false negatives, total flags, and number needed to screen.

Calibration

The calibration utilities are provided in zedstat.calibration.

Held-out isotonic calibration

Given a score \(S\) and binary label \(Y \in \{0,1\}\), the calibration routine splits the data into train and test subsets. On the training subset, isotonic regression fits a monotone mapping

\begin{equation*} g(s) \approx P(Y=1 \mid S=s). \end{equation*}

The fitted function \(g\) is then applied to the held-out test scores to obtain calibrated probabilities.

Calibrated probability

The calibrated probability at score \(s\) is the local conditional event probability

\begin{equation*} m(s) = P(Y=1 \mid S=s). \end{equation*}

For continuous scores this can be interpreted as

\begin{equation*} m(s) = \frac{f_{S,Y=1}(s)}{f_S(s)} = \frac{\pi f_{S \mid Y=1}(s)}{f_S(s)}. \end{equation*}

This is a local score-level quantity.

Threshold PPV versus calibrated probability

These are different objects.

Threshold PPV at threshold \(t\) is

\begin{equation*} \mathrm{PPV}(t) = P(Y=1 \mid S \ge t) \end{equation*}

when higher scores indicate higher risk. This is a tail-average quantity:

\begin{equation*} \mathrm{PPV}(t) = \frac{\int_t^\infty m(s) f_S(s)\, ds} {\int_t^\infty f_S(s)\, ds} = E[m(S)\mid S\ge t]. \end{equation*}

Thus, calibrated probability is local, while threshold PPV is cumulative over all subjects beyond the threshold.

Brier score

The Brier score evaluates probability accuracy:

\begin{equation*} \mathrm{Brier} = \frac{1}{n}\sum_{i=1}^n (p_i - y_i)^2 \end{equation*}

where \(p_i\) is the predicted probability and \(y_i\) is the observed binary outcome. Lower is better, with 0 indicating perfect probabilistic prediction.

Calibration intercept and slope

A well-calibrated model should satisfy

\begin{equation*} P(Y=1 \mid \hat p = p) = p. \end{equation*}

A practical assessment regresses the observed outcome on the logit of the predicted probability:

\begin{equation*} \logit P(Y=1) = a + b \logit(\hat p). \end{equation*}

Here:

  • \(a\) is the calibration intercept. Ideal value is 0.

  • \(b\) is the calibration slope. Ideal value is 1.

An intercept above 0 indicates underprediction on average. An intercept below 0 indicates overprediction on average. A slope below 1 indicates overly extreme predictions; a slope above 1 indicates predictions compressed toward the center.

Calibration table and reliability diagram

Predicted probabilities are grouped into bins, and for each bin the observed event rate is estimated. If a bin contains \(k\) events among \(n\) subjects, the observed rate is

\begin{equation*} \hat p_{\mathrm{obs}} = \frac{k}{n}. \end{equation*}

Wilson confidence intervals are computed for each bin and displayed as vertical error bars in the reliability diagram.

Sample size planning

The sample size utilities in zedstat use AUC-based approximations. Let the target AUC be \(A\). Define

\begin{equation*} Q_1 = \frac{A}{2-A}, \qquad Q_2 = \frac{2A^2}{1+A}. \end{equation*}

In the balanced-design approximation, the required sample size per class for resolving an AUC tolerance \(\delta\) at confidence level \(1-\alpha\) is

\begin{equation*} n \approx \frac{z_{1-\alpha/2}^2 \, c}{\delta^2}, \end{equation*}

where

\begin{equation*} c = A(1-A) - A^2 + Q_1 + Q_2. \end{equation*}

When prevalence is specified, the code uses a prevalence-aware total sample size formula derived from the Hanley-McNeil variance approximation.

Remarks

zedstat separates two different but complementary notions of risk:

  • threshold-level decision utility, such as PPV, NPV, and likelihood ratios, derived from the ROC curve and prevalence;

  • score-level probability interpretation, obtained through calibration.

The first is useful for screening policy and operating-point selection. The second is useful when the score must be interpreted as an individual event probability.

Metadata

Release files for zedstat 0.2.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for zedstat 0.2.3
File Size Uploaded Provenance Actions
zedstat-0.2.3.tar.gz 133.6 kB Without provenance Details for zedstat-0.2.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for zedstat 0.2.3
File Interpreter ABI Platform Actions
zedstat-0.2.3-py3-none-any.whl Python 3 none any Details for zedstat-0.2.3-py3-none-any.whl

Total release size: 350.4 kB

Release files / zedstat-0.2.3.tar.gz

Download URL zedstat-0.2.3.tar.gz
Size 133.6 kB
Tags Source
SHA-256 checksum
How to use checksums
26625e0ddecf5ef5d7f4e73421d9453d977613ecd896e37a8d5ea73fcc20dcf8
BLAKE2b-256 checksum
How to use checksums
6090cacc768ec8bdaf7d573a7a51352c378fed1505dc708c411e9add92db03d7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.7

Release files / zedstat-0.2.3-py3-none-any.whl

Download URL zedstat-0.2.3-py3-none-any.whl
Size 216.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
5913f14715679ea0e33226fcd4f50d71ea8387b96b8449598ed3401116a46d17
BLAKE2b-256 checksum
How to use checksums
d2204be84cc413daf5ec2299259a70a6e2654a5e755aed71e74bccef3ec462f3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.7

Security for zedstat 0.2.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.3 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page