PrivateLabBench

Evaluate scientific models on private data, from your terminal.

Quickstart · Prediction tables · Receipts · Contributing

PrivateLabBench is a local evaluation framework for scientific machine learning. Bring a prediction table, measure performance across datasets and subgroups, estimate uncertainty, and compare models on matched samples. Keep the underlying data in your environment and exchange aggregate evaluation receipts with collaborators.

Evaluation runs locally without an account, hosted service, or data upload.

Installation

Use Python 3.10 or later. Create an environment for v0.1.0:

python -m venv .venv

Activate the environment and install:

# macOS / Linux
source .venv/bin/activate

# Windows PowerShell
.venv\Scripts\Activate.ps1

python -m pip install private-lab-bench==0.1.0
plb --help

The privatelabbench command is also available. Optional molecular featurization uses python -m pip install 'private-lab-bench[rdkit]==0.1.0'. See the installation guide for source/development installs, or run the quickstart notebook in Colab.

Run Your First Evaluation

Start with a bundled synthetic benchmark pack:

plb list-packs
plb run-pack community-tabular-regression@1.0.0
plb verify reports/benchmark_packs/tabular_regression_receipt.shareable.json

Packs include example predictions and versioned evaluation settings. They work from installed wheels and require no model downloads or training. Other packs cover molecular regression and protein binary classification.

Evaluate Your Predictions

Use prediction-table/v1 to evaluate outputs from your own model:

sample_id,target,prediction,site,batch
s01,0.12,0.14,lab-a,b1
s02,0.20,0.23,lab-a,b1
s03,0.33,0.31,lab-b,b2
s04,0.41,0.45,lab-b,b2

Save the table as predictions.csv, then run:

plb evaluate predictions.csv --slice site --slice batch
plb verify reports/evaluation/evaluation_receipt.json

Regression, binary classification, and multiclass probability tables are supported. Ambiguous columns or task types require explicit flags; binary probabilities with named labels require ordered negative/positive classes. Results stay local until a reviewed release policy permits export. This four-row example is too small for the default bootstrap interval or slice minimum. See the 5-minute quickstart for defaults and the prediction-table reference for classification schemas.

Make Runs Reproducible

Store task, input, slice, uncertainty, and reporting settings in YAML:

plb init predictions.csv --slice site --out evaluation.yaml
plb run evaluation.yaml
plb validate-config configs/tabular_uncertainty_eval.yaml
plb run configs/tabular_uncertainty_eval.yaml

Config runs write local JSON and Markdown reports, an artifact manifest, and evaluation receipts. Review the uncertainty settings and choose a sampling method appropriate to your data. See statistical uncertainty for assumptions and limits.

Use plb list-protocols and plb run local.yaml --protocol plb://generic/regression@1.0.0 to apply fixed, versioned evaluation rules across private datasets. Receipts record the exact protocol digest. See evaluation protocols for local definitions, package providers, and release controls.

Compare Models

Compare two prediction tables containing the same sample IDs and targets:

plb compare model-a.csv model-b.csv --target target --task-type regression --metric rmse --resamples 1000 --practical-threshold 0.01

Paired comparisons report metric differences, bootstrap intervals, and a decision relative to your practical threshold. Optional slice comparisons help locate differences across groups. See the paired-comparison guide.

Combine compatible shared receipts with plb aggregate lab-a.shareable.json lab-b.shareable.json --model model@revision. The aggregation guide covers site identities, signer verification, and the metrics that support pooled point estimates.

Inspect cohort changes with plb evaluate current.csv --reference reference.csv --slice site. Reference/current diagnostics report performance degradation, slice prevalence, and selected distribution changes, with explicit uncertainty and release controls.

Track the same model over repeated evaluations with plb trend. It compares verified shareable receipts at declared collection times, retains uncertainty and quality decisions, and reports changing slices without needing prior raw datasets.

For repeated or nested observations, declare a sampling design to resample independent clusters and keep paired models aligned. Optional slice-test families record dependence assumptions and multiplicity adjustments.

Check declared split integrity before scoring with plb audit-design or plb evaluate --evaluation-design design.yaml. Evaluation-design audits detect supported overlap/group/time violations and keep unavailable training evidence explicit.

Evaluate real Caco2 and PGP benchmark predictions with the optional TDC ADMET integration. It preserves the pinned official test sets and metrics, checks upstream score agreement, and supports the same evaluation protocols on private ADMET tables.

The Matbench integration supports composition and structure tasks with pinned official folds, units, corrected probability ROC-AUC, and verified upstream fold aggregation.

The ProteinGym integration evaluates DMS substitution scores within each assay, with tied-rank Spearman and explicit protein/function aggregation.

Describe evaluations of released models with the external-validation profile: immutable model identity, declared evaluator relationship and dataset provenance, plus explicitly approved aggregate receipts that can be compared across labs.

When predictions need to be generated inside the lab, use the trusted local scorer with a reviewed numeric model, pinned digest, and explicit feature units. Its optional evaluation pipeline reuses the normal protocol and release controls.

Operate a hidden benchmark with owner-held labels, strict prediction submissions, approved result receipts, and a verified results table. Ranking requires an explicit rule in the pinned scientific protocol.

Inspect Evaluation Receipts

Follow the Caco2 model-selection case from a fresh environment to a paired decision, critical-group diagnosis, release check, and signed receipt. It distinguishes real public benchmark evidence from its synthetic measurement-shift proxy; independent researcher trials remain pending.

evaluation-receipt/v1 separates local context from aggregate information intended for exchange:

plb verify reports/tabular_uncertainty_demo_receipt.shareable.json

Shareable receipts require an explicit release-policy/v1 policy. A blocked export leaves local results and a release audit, and removes any previous shareable file. The bundled synthetic packs include reviewed example policies; private-data runs need their own review. Use plb release-check <local-receipt.json> --policy release.yaml for a dry run.

Verification checks artifact integrity and optional signatures. A matching self-contained hash alone does not establish authorship, correct execution, or scientific validity. See evaluation receipts and report integrity.

For offline signer authentication, use optional Ed25519 signatures and pinned public keys:

plb verify receipt.shareable.json --trust-store trusted-labs.yaml --require-trusted-signer --json

See the receipt trust guide for signing, key rotation, protocol binding, and the limits of these checks.

Privacy and Sharing

Keeping data local limits data movement; aggregate results can still disclose sensitive information. PrivateLabBench provides separate tools for exact reporting, empirical privacy audits, release-policy checks, and heuristic metric_perturbation.

Metric perturbation, including the deprecated dp alias, does not provide a formal differential privacy guarantee. Bounded-query DP reference primitives have their own sensitivity and accounting requirements. Read the privacy guide before selecting a release method.

Reference

Enforce explicit model thresholds in CI with plb gate <receipt> --policy quality.yaml. See quality gates for confidence, baseline, slice, and privacy requirements.

Use --help to explore commands and options:

plb eval-predictions --help
plb compare --help
plb list-tasks
plb list-privacy-attacks

• Five-minute quickstart
• Core concepts
• Tasks and plugins
• Community benchmark packs
• Troubleshooting
• Project scope and roadmap

Developing

Install the development and documentation tools:

python -m pip install -e '.[dev,docs]' build twine
python -m pytest -q
python -m build
python -m twine check dist/*
mkdocs build

CI tests Python 3.10, 3.11, and 3.12, validates packages and documentation, and exercises the Docker runner. It also installs the wheel in a clean environment and runs a bundled pack outside the checkout. Release publishing uses the tag-based release workflow.

Useful source entry points:

Contributing

Focused fixes, scientific-domain plugins, benchmark packs, and documentation improvements are welcome. Read CONTRIBUTING.md for setup, extension boundaries, and validation requirements. Discuss new tasks, privacy methods, or schema changes in an issue before implementation.

The project follows its Code of Conduct and governance guide. Report sensitive vulnerabilities through the process in SECURITY.md.

License

Released under the MIT license. Use CITATION.cff to cite PrivateLabBench in research.