PrivateLabBench

Evaluate scientific models on private data, from your terminal.

Quickstart · Documentation · v0.1.1 release · Contributing

PrivateLabBench is a local evaluation framework for scientific machine learning. Bring model predictions and a validation table to compare candidates, estimate uncertainty, and find failures across sites, assays, or other groups. Keep the underlying data in your environment and share approved aggregate receipts with collaborators.

Evaluation runs without an account, hosted service, or data upload. A receipt records the evaluation protocol, selected results, and the evidence available to verify them.

Installation

The current release is v0.1.1, available on PyPI and GitHub Releases. CI tests Python 3.10, 3.11, and 3.12.

Create a virtual environment:

python -m venv .venv

Activate it with source .venv/bin/activate on macOS/Linux, or .venv\Scripts\Activate.ps1 in Windows PowerShell. Then install:

python -m pip install private-lab-bench==0.1.1
plb --help

The longer command name, privatelabbench, works too. See the installation guide for optional dependencies and source installs, or open the quickstart in Colab.

Run a first evaluation

Run a bundled synthetic example in a fresh directory and verify its shareable receipt:

mkdir plb-example
cd plb-example
plb list-packs
plb run-pack community-tabular-regression@1.0.0
plb verify reports/benchmark_packs/tabular_regression_receipt.shareable.json

The pack includes predictions, evaluation settings, and an example disclosure policy. It runs from the installed package without a checkout or model download. Other packs cover molecular regression and protein binary classification. Pack and protocol versions are independent of the package release.

Evaluate your predictions

Save a table like this as predictions.csv:

sample_id,target,prediction,site
s01,0.12,0.14,lab-a
s02,0.20,0.23,lab-a
s03,0.33,0.31,lab-b
s04,0.41,0.45,lab-b

plb evaluate predictions.csv --task-type regression --slice site
plb verify reports/evaluation/evaluation_receipt.json

This writes local reports and a receipt. The four rows illustrate the schema; the default bootstrap requires at least 20 observations, and slice metrics require at least five per group. Repeated patients, compounds, or batches need a declared sampling design.

Binary classification and multiclass probability tables are also supported. Class order and probability meaning matter: use explicit labels when needed. The prediction-table reference covers column bindings, missing values, and classification examples.

Save the settings for another run:

plb init predictions.csv --task-type regression --slice site --out evaluation.yaml
plb validate-config evaluation.yaml
plb run evaluation.yaml

Use a versioned evaluation protocol when collaborators need the same metrics, sampling rules, and decision criteria. Receipts bind its identity and digest.

Compare models

Provide one prediction table per model, with identical sample IDs and targets:

plb compare model-a.csv model-b.csv --target target --task-type regression --metric rmse --practical-threshold 0.01

The comparison reports the paired improvement, bootstrap interval, and whether the evidence supports your practical threshold. Subgroup comparisons help locate the difference. See the paired-comparison guide for noninferiority, sampling assumptions, and interval interpretation.

Scientific workflows

The initial benchmark integrations reproduce pinned upstream scoring rules:

The Caco2 case study follows a model-selection decision through uncertainty, critical-group checks, approved export, and signature verification. It distinguishes the real public control from a synthetic measurement-shift proxy. Independent researcher onboarding trials remain pending.

Receipts and sharing

A reviewed release policy controls export. When export is blocked, the run keeps its local results and release audit and removes a previous shareable receipt. Bundled packs use policies for synthetic examples; private-data evaluations need their own review.

plb release-check local_receipt.json --policy release.yaml
plb verify receipt.shareable.json --trust-store trusted-labs.yaml --require-trusted-signer --json

Verification distinguishes artifact integrity from an authenticated signer statement. Execution and scientific validity require separate evidence. The receipt trust guide covers signing and pinned keys.

Keeping rows local limits data movement, but aggregates can still disclose sensitive information. Empirical audits, disclosure policies, and heuristic metric perturbation address different questions. Metric perturbation, including the deprecated dp alias, provides no formal differential privacy guarantee. Read the privacy guide before sharing private results.

Developing and contributing

git clone https://github.com/dipeshbabu/private-lab-bench.git
cd private-lab-bench
python -m pip install -e '.[dev,docs]' build twine
python -m pre_commit install
python -m pre_commit run --all-files
python -m pytest -q
python -m mkdocs build --strict

CI checks supported Python versions, package builds, installation outside the checkout, documentation, Docker, scientific adapters, and repository history. Read CONTRIBUTING.md for extension boundaries and validation requirements.

Report bugs or propose tasks through GitHub Issues. Use the process in SECURITY.md for sensitive vulnerabilities. Contributions follow the Code of Conduct and governance guide.

License and citation

PrivateLabBench uses the MIT license. Cite the release or commit you used with CITATION.cff.