cell-eval

Description

This package provides a comprehensive suite of metrics for evaluating the performance of models that predict cellular responses to perturbations at the single-cell level. It can be used either as a command-line tool or as a Python module.

Installation

Distribution with uv

# install from pypi
uv pip install -U cell-eval

# install from github directly
uv pip install -U git+https://github.com/arcinstitute/cell-eval

# install cli with uv tool
uv tool install -U git+https://github.com/arcinstitute/cell-eval

# Check installation
cell-eval --help

Usage

To get started you'll need to have two anndata files.

1. a predicted anndata (adata_pred).
2. a real anndata to compare against (adata_real).

Prep (VCC)

To prepare an anndata for VCC evaluation you can use the cell-eval prep command. This will strip the anndata to bare essentials, compress it, adjust naming conventions, and ensure compatibility with the evaluation framework.

This step is optional for downstream usage, but recommended for optimal performance and compatibility.

Run this on your predicted anndata:

cell-eval prep \
    -i <your/path/to>.h5ad \
    -g <expected_genelist>

Run

To run an evaluation between two anndatas you can use the cell-eval run command.

This will run differential expression for each anndata and then run a suite of evaluation metrics to compare the two (select your suite of metrics with the --profile flag).

To save time you can submit precomputed differential expression results, see the cell-eval run --help menu for more information.

cell-eval run \
    -ap <your/path/to/pred>.h5ad \
    -ar <your/path/to/real>.h5ad \
    --num-threads 64 \
    --profile full

To run this as a python module you will need to use the MetricsEvaluator class.

from cell_eval import MetricsEvaluator
from cell_eval.data import build_random_anndata, downsample_cells

adata_real = build_random_anndata()
adata_pred = downsample_cells(adata_real, fraction=0.5)
evaluator = MetricsEvaluator(
    adata_pred=adata_pred,
    adata_real=adata_real,
    control_pert="control",
    pert_col="perturbation",
    num_threads=64,
)
(results, agg_results) = evaluator.compute()

This will give you metric evaluations for each perturbation individually (results) and aggregated results over all perturbations (agg_results).

Data ceiling

To estimate the maximum achievable score on each metric given the noise inherent in the real data, pass --ceiling. This is computed from the real data only: each perturbation's cells (and the control's) are split into two disjoint halves of n/2 cells (no cell in both), one half plays "real" and the other "prediction", and the full metric suite is run on that self-split. Averaging each metric over perturbations and applying the analytical Spearman-Brown correction r' = 2r/(1+r) maps that per-context mean from half depth back to full depth. The result is, per metric, an unbiased upper bound on how well any model could score on this dataset.

A disjoint split is used rather than a bootstrap self-split: a bootstrap draws the two halves from the same cells, so they are not independent, which biases the ceiling in both directions (so it is not a reliable upper bound). The shared cells make the halves agree more than two independent samples would (inflating it), while the duplicate cells over-call the FDR-gated DE metrics and drag the recovery metrics (recall / overlap / AUC) down. The disjoint split is unbiased but shallow (each half n/2), which the Spearman-Brown doubling corrects.

The correction is applied only to a fixed set of reliability metrics (the SB_METRICS list in _evaluator.py); every other metric — error metrics, unbounded counts, and reliability metrics left off that list (clustering_agreement, pearson_edistance) — is reported as NaN.

cell-eval run \
    -ap <your/path/to/pred>.h5ad \
    -ar <your/path/to/real>.h5ad \
    --num-threads 64 \
    --profile full \
    --ceiling

This is additive: it writes the normal results.csv / agg_results.csv and ceiling_results.csv (the raw per-perturbation self-split) / agg_ceiling_results.csv (the SB-corrected per-metric ceiling). The split is reproducible via --ceiling-seed (default 0). From python, call compute_ceiling on the evaluator:

ceiling, ceiling_agg = evaluator.compute_ceiling(seed=0)

Score

To normalize your scores against a baseline you can run the cell-eval score command.

This accepts two agg_results.csv (or agg_results objects in python) as input.

cell-eval score \
    --user-input <your/path/to/user>/agg_results.csv \
    --base-input <your/path/to/base>/agg_results.csv

Or from python:

from cell_eval import score_agg_metrics

user_input = "./cell-eval-user/agg_results.csv"
base_input = "./cell-eval-base/agg_results.csv"
output_path = "./score.csv"

score_agg_metrics(
    results_user=user_input,
    results_base=base_input,
    output=output_path,
)

Library Design

The metrics are built using the python registry pattern. This allows for easy extension for new metrics with a well-typed interface.

Take a look at existing metrics in cell_eval.metrics to get started.

Development

This work is open-source and welcomes contributions. Feel free to submit a pull request or open an issue.

Citation

Any publication that uses this source code should cite the State paper.