SurvivalEVAL 0.8.7

SurvivalEVAL is a Python 3.10+ package for evaluating survival analysis predictions. It supports right-censored and interval-censored outcomes, predicted survival curves, point predictions, single-time probabilities, and quantile-regression outputs.

The package is designed around evaluator classes:

• SurvivalEvaluator: general right-censored survival-curve predictions.
• LifelinesEvaluator, PycoxEvaluator, and ScikitSurvivalEvaluator: adapters for common model-output formats.
• IntervalCenEvaluator: interval-censored survival-curve predictions.
• PointEvaluator: point survival-time predictions.
• SingleTimeEvaluator: probabilities at one target time.
• QuantileRegEvaluator: predicted event-time quantiles.

The public tests and examples show typical model integrations. See examples for notebooks covering lifelines, pycox, scikit-survival, quantile prediction, point prediction, interpolation choices, and monotonicity handling.

Metric Guide

SurvivalEVAL groups metrics by prediction target:

• Point prediction metrics compare predicted event times with observed event/censoring times.
• Single-time probability metrics evaluate survival or event probability at one target time.
• Survival distribution metrics evaluate the full predicted survival curve.
• Interval-censored metrics evaluate survival curves against observed event intervals.

Installation

Install from PyPI:

pip install SurvivalEVAL

For local development:

git clone https://github.com/shi-ang/SurvivalEVAL.git
cd SurvivalEVAL
python -m pip install -r requirements.txt
python -m pip install -e .

Optional development dependencies are available with:

python -m pip install -e ".[dev]"

For a reproducible development environment using the committed uv.lock:

uv sync --locked --extra dev

To update dependencies, run uv lock --upgrade, sync the environment, and run the tests before committing pyproject.toml, requirements.txt, and uv.lock together when they change. Explicit security minimums also apply to pip installations; the lockfile itself only controls uv project environments. Dependabot checks for uv dependency updates weekly.

Input Conventions

For right-censored data, event_indicators uses:

• 1: observed event
• 0: right-censored observation

For interval-censored data, pass finite non-negative left_limits and right_limits. Use right_limits=np.inf for right-censored observations. Exact events can be represented with left_limits == right_limits, and left-censored observations can use left_limits == 0.

Predicted survival curves are passed as survival probabilities over time:

• pred_survs: shape (n_samples, n_time_points) or (n_time_points,)
• time_coordinates: shape (n_time_points,) or (n_samples, n_time_points)

At least one of pred_survs or time_coordinates must be two-dimensional so the evaluator can infer the number of testing samples. If the time grid does not start at zero, SurvivalEVAL prepends time zero and survival probability one.

Quickstart: Right-Censored Survival Curves

from lifelines import CoxPHFitter
from lifelines.datasets import load_rossi

from SurvivalEVAL import LifelinesEvaluator

rossi = load_rossi().sample(frac=1.0, random_state=0)
train = rossi.iloc[:300, :]
test = rossi.iloc[300:, :]

cph = CoxPHFitter()
cph.fit(train, duration_col="week", event_col="arrest")

survival_curves = cph.predict_survival_function(test)

evl = LifelinesEvaluator(
    survival_curves,
    test.week.values,
    test.arrest.values,
    train.week.values,
    train.arrest.values,
)

c_index, concordant, total = evl.concordance(method="Harrell")
td_c_index, td_concordant, td_total = evl.concordance_time_dependent(method="Antolini")

mae = evl.mae(method="Pseudo_obs")
ibs = evl.integrated_brier_score(num_points=53)
d_cal_p, d_cal_hist = evl.d_calibration()

auc = evl.auc(target_time=25)
brier = evl.brier_score(target_time=25)
one_cal_p, observed, expected = evl.one_calibration(target_time=25)

Quickstart: Interval-Censored Survival Curves

import numpy as np

from SurvivalEVAL import IntervalCenEvaluator

time_grid = np.array([0.0, 1.0, 2.0, 3.0, 4.0])
pred_survs = np.array(
    [
        [1.0, 0.82, 0.58, 0.34, 0.12],
        [1.0, 0.76, 0.51, 0.25, 0.08],
        [1.0, 0.90, 0.72, 0.48, 0.22],
    ]
)

left = np.array([0.5, 1.0, 2.5])
right = np.array([1.5, np.inf, 3.5])

train_left = np.array([0.0, 1.0, 1.5, 2.0])
train_right = np.array([1.0, 2.0, np.inf, 3.0])

evl = IntervalCenEvaluator(
    pred_survs,
    time_grid,
    left,
    right,
    train_left_limits=train_left,
    train_right_limits=train_right,
)

c_index, concordant, total = evl.concordance(method="comparable")
brier = evl.brier_score(target_time=2.0, method="uncensored")
ibs = evl.integrated_brier_score(
    target_times=np.array([1.0, 2.0, 3.0]),
    method="uncensored",
)

one_cal_p, observed, expected = evl.one_calibration(
    target_time=2.0,
    method="MidPoint",
)
d_cal_p, d_cal_hist = evl.d_calibration()
coverage, coverage_gap, average_width = evl.coverage(
    cov_level=0.8,
    method="linear",
)

Right-Censored Metrics

Right-censored survival-curve evaluators include SurvivalEvaluator, LifelinesEvaluator, PycoxEvaluator, ScikitSurvivalEvaluator, and QuantileRegEvaluator.

Point Prediction Discrimination

These metrics compare predicted survival times with observed event/censoring times. For survival-curve evaluators, predicted times are derived using the configured predict_time_method: "Median", "Mean", or "RMST".

"Uno", "IPCW", and "Margin" require training event times and indicators. When tau is omitted, no concordance truncation is applied.

Point Prediction Errors And Reliability

MAE, MSE, and RMSE share the same censoring-handling methods. Use evl.mae(method=...), evl.mse(method=...), or evl.rmse(method=...).

"Margin", "IPCW-T", "IPCW-D", and "Pseudo_obs" require training event times and indicators. Hinge uses training data only when weighted=True.

Single-Time Probability Metrics

These metrics evaluate survival probabilities at a specified target time. Use SingleTimeEvaluator when the model only outputs one survival probability per patient at one target time.

Survival Distribution Metrics

These metrics evaluate the full predicted survival curve.

For time-dependent concordance, risks="Survival" uses -S(t | z) as the risk score at each event anchor, while risks="Hazard" uses estimated hazard rates directly. method="IPCW" requires training event times and indicators. tau keeps event anchors whose observed time is strictly before tau; when omitted, no truncation is applied.

The evaluator predicts each sample's curve at most once and counts comparisons by ranking risks within equal-event-time groups. Counting uses O(n) working memory and O(n log n + n U log(M + 1)) time, where U is the number of contributing event times and M is the largest event group. This avoids visiting every pair when many events share a time; it remains quadratic when all event times differ. With mostly distinct event times, rank-query overhead can outweigh the savings. Interpolation adds O(k) temporary storage for a curve with k grid points. These bounds exclude the input curves and IPCW training data. The lower-level concordance_time_dependent function still accepts a dense sample-by-event risk matrix and counts its pairs directly, since separate columns at the same event time may contain different scores.

Interval-Censored Metrics

IntervalCenEvaluator evaluates predicted survival curves against interval endpoints. It supports exact events, left-censored observations, interval censoring, and right censoring within one interface.

Interval-Censored Discrimination

Interval-Censored Error And Scoring Metrics

The "Tsouprou-conditional" method requires test and train covariates through x and x_train.

Interval-Censored Calibration

Other Evaluators And Helper APIs

SurvivalEVAL.Evaluations.OtherMetrics includes research helpers such as calibration slope and coefficient of variation.

Nonparametric Estimators

The SurvivalEVAL.NonparametricEstimator.SingleEvent module includes:

For censoring survival, use SingleEvent.KaplanMeier(times, event_indicators, reverse=True) with the original event indicators (1 = observed event, 0 = censored). Original events leave the risk set before tied censorings. IPCW metrics use this reverse KM estimate and evaluate right-continuous G(t), including the censoring update at time t.

Citing This Work

We recommend you use the following to cite SurvivalEVAL in your publications:

@article{qi2024survivaleval,
year = {2024},
month = {01},
pages = {453-457},
title = {{SurvivalEVAL}: A Comprehensive Open-Source Python Package for Evaluating Individual Survival Distributions},
author = {Qi, Shi-ang and Sun, Weijie and Greiner, Russell},
volume = {2},
journal = {Proceedings of the AAAI Symposium Series},
doi = {10.1609/aaaiss.v2i1.27713}
}