prs-pipeline

Dagster pipeline for computing PRS reference distributions from the 1000 Genomes reference panel.

Overview

This pipeline downloads the PGS Catalog 1000G reference panel (~7 GB), computes polygenic risk scores for all 2,504 reference individuals across all PGS Catalog scores, aggregates per-superpopulation distribution statistics, and pushes reference_distributions.parquet to HuggingFace (just-dna-seq/prs-percentiles).

End users of just-prs automatically pull this tiny parquet via PRSCatalog.reference_distributions().

Running

cd prs-pipeline
uv run dagster dev -m prs_pipeline.definitions

Then open http://localhost:3000 in your browser.

Assets

Scoring & Distribution Pipeline

Asset	Group	Description
`ebi_reference_panel_fingerprint`	download	HTTP fingerprint for freshness tracking of the remote reference panel
`ebi_scoring_files_fingerprint`	download	HTTP fingerprint for the remote scoring file manifest
`scoring_files`	download	Bulk-download all harmonized PGS scoring `.txt.gz` files from EBI FTP
`scoring_files_parquet`	compute	Convert all `.txt.gz` scoring files to spec-driven parquet caches (zstd-9, embedded headers). Deletes `.txt.gz` after verified conversion to save ~5.5 GB disk space. Tracks per-file failures in `conversion_failures.parquet`
`reference_panel`	download	Download + extract reference panel binary files (.pgen/.pvar/.psam)
`reference_scores`	compute	Score all PGS IDs against the reference panel via `compute_reference_prs_batch()`
`reference_percentile_audit`	compute	Audit cached or HuggingFace reference percentile parquets, log pass/warn/fail counts, and write/upload quality sidecars without recomputing scores
`public_sample_score_parts`	compute	PGS-major unrestored then restored scoring of `--vcf` genomes into atomic checkpoint parts. Scores ≥1M variants are singleton checkpoints; DuckDB joins them in bounded slices; workers recycle at the env memory budget or after a large score. Does not recompute 1000G scores
`public_sample_runtime_results`	compute	Compact checkpoint parts once into `runtime_results.parquet` and enforce the complete sample×PGS×profile outcome matrix
`public_sample_canary_audit`	compute	Derive public canary rows from unrestored `public-wgs-pass-v1` runtime; score unknown/private `--vcf` separately; write catalog flags and percentile audit sidecars
`hf_public_sample_runtime`	upload	Atomic allowlisted commit of runtime-owned files (`samples.parquet`, `runtime_results.parquet`, `runtime_manifest.json`, `sample_ancestry.parquet`) plus flags/sidecars. Never uploads evidence, root docs, identity cache, or checkpoint parts
`sample_score_evidence`	compute	Catalog-level evidence tables from one pinned catalog snapshot. Writes `evidence_manifest.json`. Does not score genomes or write final docs
`hf_sample_score_evidence`	upload	Atomic allowlisted commit of the nine evidence parquets plus `evidence_manifest.json`. Never uploads runtime scores or root README/AGENTS
`sample_score_integration`	compute	Stage pinned runtime/evidence/catalog/percentile snapshots and build `model_analysis.parquet` plus ancestry-selected `trait_summaries.parquet` and final docs. Does not rescore genomes
`hf_sample_score_dataset`	upload	Atomic six-path commit of model_analysis, trait_summaries, final manifest.json, README.md, AGENTS.md, and ANALYSIS.md
`hf_prs_percentiles`	upload	Enrich distributions with metadata and absolute risk, push to HuggingFace

Metadata & Prevalence Pipeline

Asset	Group	Description
`raw_pgs_metadata`	download	Download PGS Catalog bulk metadata sheets (scores, performance, evaluation, publications)
`cleaned_pgs_metadata`	compute	Clean and normalize metadata, produce parquets including `publications.parquet`
`gwas_studies`	download	Download GWAS Catalog bulk studies + trait mappings, parse case/control from free-text
`trait_prevalence`	compute	Merge 3-tier prevalence data (seed CSV → GWAS cohorts → PGS eval cohorts) into `trait_prevalence.parquet`
`hf_pgs_catalog`	upload	Push cleaned metadata + prevalence to HuggingFace

For details on how absolute risk estimation works, see the methodology document.

Metadata

Release files for prs-pipeline 0.2.12

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for prs-pipeline 0.2.12
File	Size	Uploaded	Provenance
prs_pipeline-0.2.12.tar.gz	66.7 kB	Sep 3, 2026	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for prs-pipeline 0.2.12
File	Interpreter	ABI	Platform	Reset
prs_pipeline-0.2.12-py3-none-any.whl 69.8 kB Sep 3, 2026 Without provenance	Python 3	none	any	Details

Total release size: 136.5 kB

Release files / prs_pipeline-0.2.12.tar.gz

Download URL	prs_pipeline-0.2.12.tar.gz
Size	66.7 kB
Tags	Source
SHA-256 checksum How to use checksums	`02d7be0a8c1017559fdd6e419e06b8f14f2bc995f9b6c1ab10007d7a1ad0cb9e`
BLAKE2b-256 checksum How to use checksums	`16716a45a8197c3d0afd4dc5da9ed79e1db43d56bb0761796f40f4a18f0d9770`
Upload date	Sep 3, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.10.0 {"installer":{"name":"uv","version":"0.10.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}`

Release files / prs_pipeline-0.2.12-py3-none-any.whl

Download URL	prs_pipeline-0.2.12-py3-none-any.whl
Size	69.8 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`2c6976d49affe6f0f764f71ee7beb4e5fcf91006c87d308a4d6734b8695dc04a`
BLAKE2b-256 checksum How to use checksums	`d63cafe4259039975d4c254d43b210282f818457b240ee748e5ae8265924e394`
Upload date	Sep 3, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.10.0 {"installer":{"name":"uv","version":"0.10.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}`

Security for prs-pipeline 0.2.12

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.