Skip to main content

appfl-bio-suite

PyPI

Cross-silo federated learning experiments in computational biology, built on APPFL, hivewatch and Globus Compute.

Institutions train on their own data, on their own hardware, and return only results. The standard multi-user deployment uses individual identities, with no shared credential, VPN, or inbound network access.

Experimental: single-user service-account onboarding. An opt-in prototype uses one service identity per site, with credentials held by that site and the coordinator. It is not yet the default or a production-ready replacement for multi-user endpoints. See the trial guide and design evaluation.

→ mfjoel01.github.io/appfl-bio-suite — the project site, with a live preview of the federation network map.

    YOU (coordinator)                          PARTNER SITES
    ┌──────────────┐
    │   driver     │ ──── task ────►    their cluster, their data
    │  aggregator  │ ◄─── result ───    (weights, or summary statistics)
    └──────────────┘

The experiments

Fine-mapping also has a completed optional MultiSuSiE coordinator pipeline and matched-filter comparison against SuSiEx. Use run fine-mapping --backend susiex or --backend multisusie with either the local or remote driver; --results-dir keeps their outputs separate. The comparison report and figures cover both methods across cohorts and filter policies.

Four experiments, one toolchain. The first three dispatch to partner sites; the fourth is in development.

What it is Data
FLamby Fed-Heart-Disease Multi-round FedAvg on a public tabular benchmark with natural hospital splits Partners download it; ~40 KB, CC BY 4.0
Federated GWAS Single-round summary-statistic meta-analysis with polygenic-score evaluation Simulated and distributed by the coordinator
Federated fine-mapping Single-round cross-ancestry credible-set construction from per-site second moments — provably equal to a pooled fit, not an approximation of one. Governed by GA4GH DUO, DRS, TRS and TES Simulated and distributed by the coordinator
Federated methylation · work in progress Multi-round FedAvg for methylation-based subtype classification on sparse, nanopore-like input, against pooled and local-only training. Currently a single-machine simulation Synthetic, or open TCGA-LAML methylation data from the NCI GDC

Federated methylation is a work in progress. All of its sites are simulated in one process today, so there is no partner bundle or endpoint for it yet. Its results so far, including one real-data run, are recorded as measured in its record of runs.

Quickstart

Python 3.12 is required. Install the release from PyPI, with the release's pinned constraints:

python3.12 -m venv .venv && source .venv/bin/activate
python -m pip install "appfl-bio-suite[all]==0.3.1" \
    -c https://raw.githubusercontent.com/mfjoel01/appfl-bio-suite/v0.3.1/constraints.txt

Always pin the exact version. Every site in a federation must run the same release, and -c pins the rest of the dependency tree to the versions that release was tested with.

Prove the install works — a complete federation on one machine, no partners, no network, and nothing to fill in first:

appfl-bio-suite simulate gwas --scenario ci-tiny --out /tmp/gwas-ci
appfl-bio-suite run gwas --config loopback --driver serial --data-root /tmp/gwas-ci

Results land in local/output/gwas-loopback/ under the current directory. This is the same pair of commands CI runs.

Then stand up a real federation. The example federation file, the fine-mapping binary installers under scripts/, and the example configurations are not part of the wheel, so take them from the matching release tag:

git clone --branch v0.3.1 https://github.com/mfjoel01/appfl-bio-suite.git
cd appfl-bio-suite
cp federation.yaml.example local/federation.yaml
$EDITOR local/federation.yaml
appfl-bio-suite preflight

To modify the suite, install from that checkout instead: pip install -e ".[all]" -c constraints.txt. Fine-mapping with the SuSiEx backend also needs two vendored binaries on PATH; the guide below covers both install routes.

→ docs/coordinator/new-federation.md takes you from here to a running experiment across institutions you recruit yourself.

The CLI

appfl-bio-suite preflight [--experiment X]          environment, pins, configs, endpoints
appfl-bio-suite endpoint status [SITE]              is it online
appfl-bio-suite endpoint smoke [SITE]               round-trip: does it actually work
appfl-bio-suite identity validate <mapping.json>    catch mapping bugs offline, in a second
appfl-bio-suite simulate EXPERIMENT --scenario NAME  generate and split synthetic data
appfl-bio-suite run EXPERIMENT                      run it
appfl-bio-suite partner-bundle EXPERIMENT --site X  a partner's complete setup package
appfl-bio-suite watch build | serve | export        the federation, on a map
appfl-bio-suite ga4gh duo | drs | trs | tes         data use, data objects, tools, tasks

How it works

One config file. federation.yaml declares who you are, who your partners are, and what each is doing. Every generator, config, document template and check reads from it. Nothing in src/ contains a coordinator identity, an endpoint UUID, or a site name, and CI verifies that.

Generated partner bundles, not templates. partner-bundle emits a directory with a partner's configuration fully resolved — their site ID, their data assignment, your identity, their scheduler's provider block — plus their two setup documents with no placeholders left to interpret.

A closed set for partner-facing content. Bundles are built from docs/partner/ and nowhere else, enforced in code, so nothing outside that tree can reach a partner.

Exact version pinning. Every site runs byte-identical versions of the Globus Compute stack. Skew does not fail at install time; it fails as a deserialization error several rounds into a run on somebody else's cluster. constraints.txt pins it and preflight --check pins verifies it.

Failures that name the fix. The troubleshooting registry is indexed by symptom and gives the mechanism alongside the fix.

GA4GH standards

The fine-mapping experiment is assembled out of four GA4GH specifications, each closing a hole the design otherwise had.

answers
DUO Data Use Ontology May this site's data be used for this study?
DRS Data Repository Service Exactly which bytes did a site compute over?
TRS Tool Registry Service Exactly which tool version computed them?
TES Task Execution Service Run that tool, there, on those bytes.

Together: a named tool version, pinned by checksum, runs over content-addressed data whose use terms permit the study that dispatched it. Every clause of that sentence used to be an assumption.

A site enforces its own consent code. Each bundle carries a DATA_USE.json, and the site's worker matches it against the study the coordinator declared — refusing before it opens a genotype file, on its own hardware, in the account it controls. The coordinator runs the same check offline before dispatch, but that one only saves a queue wait.

Bundles are content-addressed. A sha-256 per file and a Merkle id over the set, so "did this site run on the bundle I cut for it" is a string comparison. Without it, a site running last month's bundle produces well-formed aggregates over the wrong individuals, pools without complaint, and is wrong in no visible way.

Results name the code that made them. The tool pin covers the source of the two modules APPFL ships to workers, is verified at launch, and is written into every run's ga4gh_provenance.json alongside what each site attests to having actually done.

appfl-bio-suite ga4gh duo check --experiment fine-mapping   # will every site permit this?
appfl-bio-suite ga4gh drs register --data-root local/data/… # address the bundles
appfl-bio-suite ga4gh trs publish --out local/trs           # pin the tool
appfl-bio-suite ga4gh tes task --site X                     # what would run, as a document

All of it is opt-in per experiment: a federation.yaml with no ga4gh blocks produces byte-identical configs to the ones it produced before any of this existed.

→ docs/coordinator/ga4gh.md

The network map

Everyone in the federation — coordinator and partners alike — can open the same page: every institution on a world map, which experiments each runs, what kind of data each holds, how many samples they declared, and whether their endpoint is answering.

pip install "appfl-bio-suite[watch]==0.3.1" -c constraints.txt   # included in [all]

appfl-bio-suite watch build              # draw it from federation.yaml
appfl-bio-suite watch serve              # look at it locally
appfl-bio-suite watch export --out site/ # three static files anyone can host
appfl-bio-suite run gwas --watch         # a live run, on the same map

Built on hivewatch, APPFL's monitoring toolkit, which supplies the event schema, the map format and the viewer.

The map exists before any run does. It is drawn from federation.yaml alone, so the day the config names a partner, that partner is on the map — which is what makes it a picture of the network rather than a dashboard for one job.

Site coordinates are declared in federation.yaml, never resolved from an IP address: a partner's compute node has no outbound web access, and you already know where your partners are. A site with no coordinates is reported as unplaced rather than drawn at a guessed position. Service accounts, cluster paths, your Globus identity and (by default) full endpoint UUIDs never reach the published page.

→ docs/coordinator/network-map.md

Reproducibility

  • Every simulation run writes a manifest recording the scenario, every seed, the suite commit, input and output checksums, and package versions. simulate gwas --verify <manifest> confirms a rerun mechanically.
  • The GWAS simulation is deterministic. For a fixed seed, all 35 outputs are byte-identical, verified against recorded checksums.
  • Known limits are documented. docs/experiments/gwas/DATA.md states which input cohort is not regenerable, and what the shipped substitute does and does not reproduce.

Documentation

Organized by audience.

docs/README.md Index: which document, for whom, in what order
docs/coordinator/ Running a federation
docs/partner/ What a partner receives — a closed set
docs/experiments/ Per-experiment design, data, and runbooks

Requirements

Python 3.12. A Globus account. Partners willing to run a privileged process on their own clusters.

Not required: a shared filesystem, a VPN, matching operating systems or schedulers, or inbound network access.

Licence

MIT. See LICENSE — and note the third-party terms recorded there for APPFL, FLamby, and PGS Catalog data.

For automated partner setup, see bundle-based onboarding.

Metadata

Release files for appfl-bio-suite 0.3.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for appfl-bio-suite 0.3.1
File Size Uploaded Provenance Actions
appfl_bio_suite-0.3.1.tar.gz 1.3 MB With provenance Details for appfl_bio_suite-0.3.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for appfl-bio-suite 0.3.1
File Interpreter ABI Platform Actions
appfl_bio_suite-0.3.1-py3-none-any.whl Python 3 none any Details for appfl_bio_suite-0.3.1-py3-none-any.whl

Total release size: 2.5 MB

Release files / appfl_bio_suite-0.3.1.tar.gz

Download URL appfl_bio_suite-0.3.1.tar.gz
Size 1.3 MB
Tags Source
SHA-256 checksum
How to use checksums
2fe20548304b641829a22b8e712dd511f8395bf80abef19cb468639e55234c9f
BLAKE2b-256 checksum
How to use checksums
484848ee557cebc8eaf6f4f9bebe8fa36fab8cef84a7aa086dbacf5afe019c62
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.

Transparency log

Release files / appfl_bio_suite-0.3.1-py3-none-any.whl

Download URL appfl_bio_suite-0.3.1-py3-none-any.whl
Size 1.2 MB
Tags Python 3
SHA-256 checksum
How to use checksums
d92d0817b44dc21c46050c928a88cc523ff6caa0b505508768472d3504d17a14
BLAKE2b-256 checksum
How to use checksums
341c6b4ec1d54e3d798d8190d43c402aada6a5c8d6b43b0d01b1326d68e08f74
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.

Transparency log

Security for appfl-bio-suite 0.3.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for appfl_bio_suite-0.3.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page