prs-pipeline
Dagster pipeline for computing PRS reference distributions from the 1000 Genomes reference panel.
Overview
This pipeline downloads the PGS Catalog 1000G reference panel (~7 GB), computes polygenic risk scores
for all 2,504 reference individuals across all PGS Catalog scores, aggregates per-superpopulation
distribution statistics, and pushes reference_distributions.parquet to HuggingFace
(just-dna-seq/prs-percentiles).
End users of just-prs automatically pull this tiny parquet via PRSCatalog.reference_distributions().
Running
cd prs-pipeline
uv run dagster dev -m prs_pipeline.definitions
Then open http://localhost:3000 in your browser.
Assets
Scoring & Distribution Pipeline
| Asset | Group | Description |
|---|---|---|
ebi_reference_panel_fingerprint |
download | HTTP fingerprint for freshness tracking of the remote reference panel |
ebi_scoring_files_fingerprint |
download | HTTP fingerprint for the remote scoring file manifest |
scoring_files |
download | Bulk-download all harmonized PGS scoring .txt.gz files from EBI FTP |
scoring_files_parquet |
compute | Convert all .txt.gz scoring files to spec-driven parquet caches (zstd-9, embedded headers). Deletes .txt.gz after verified conversion to save ~5.5 GB disk space. Tracks per-file failures in conversion_failures.parquet |
reference_panel |
download | Download + extract reference panel binary files (.pgen/.pvar/.psam) |
reference_scores |
compute | Score all PGS IDs against the reference panel via compute_reference_prs_batch() |
reference_percentile_audit |
compute | Audit cached or HuggingFace reference percentile parquets, log pass/warn/fail counts, and write/upload quality sidecars without recomputing scores |
hf_prs_percentiles |
upload | Enrich distributions with metadata and absolute risk, push to HuggingFace |
Metadata & Prevalence Pipeline
| Asset | Group | Description |
|---|---|---|
raw_pgs_metadata |
download | Download PGS Catalog bulk metadata sheets (scores, performance, evaluation, publications) |
cleaned_pgs_metadata |
compute | Clean and normalize metadata, produce parquets including publications.parquet |
gwas_studies |
download | Download GWAS Catalog bulk studies + trait mappings, parse case/control from free-text |
trait_prevalence |
compute | Merge 3-tier prevalence data (seed CSV → GWAS cohorts → PGS eval cohorts) into trait_prevalence.parquet |
hf_pgs_catalog |
upload | Push cleaned metadata + prevalence to HuggingFace |
For details on how absolute risk estimation works, see the methodology document.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file prs_pipeline-0.2.9.tar.gz.
File metadata
- Download URL: prs_pipeline-0.2.9.tar.gz
- Upload date:
- Size: 53.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.9.17 {"installer":{"name":"uv","version":"0.9.17","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"22.04","id":"jammy","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
99ec307e8f21cba79e171c4eeeb71f6ed36a4e88583e05e1ed3068f6ec854791
|
|
| MD5 |
bd95ef58bb8e3f5b94480e0147d82a48
|
|
| BLAKE2b-256 |
f70c9ae302978971095244d8652aed44f2b4157c9e2202581b24fb119a907208
|
File details
Details for the file prs_pipeline-0.2.9-py3-none-any.whl.
File metadata
- Download URL: prs_pipeline-0.2.9-py3-none-any.whl
- Upload date:
- Size: 57.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.9.17 {"installer":{"name":"uv","version":"0.9.17","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"22.04","id":"jammy","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
dd3ac755fc9adbc856a4425876db980a3ab8946d1466e349419b14407b8fb5de
|
|
| MD5 |
67b566458df318f73bf838db40bf4e1c
|
|
| BLAKE2b-256 |
9fb31fa8ccd8493fd39f2930e1f6dca4c2b4ea22ba3f23e167fa3afe9d5a178c
|
