pgen-polars 0.1.0

Alpha Rust-native PLINK and BGEN reader for Python and Polars.

pgen-polars

Rust-native reader for PLINK 2 .pgen, PLINK 1 .bed, and BGEN v1.2 genotype files, with Python and Polars integration.

[!WARNING] Alpha software: pgen-polars is under active development. APIs, supported formats, and behavior are subject to change between releases. Pin the package version when using it in other projects.

Python package installation

Create or activate a Python 3.12 or newer virtual environment, then install the package from PyPI. Because the project is alpha software, pin the exact version in applications and reproducible environments:

python -m venv .venv
source .venv/bin/activate  # Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install "pgen-polars==0.1.0"

With uv:

uv add "pgen-polars==0.1.0"

Verify the installation:

python -c "import pgen_polars; print(pgen_polars.__version__)"

Published wheels support CPython 3.12 and newer on Linux (x86-64 and ARM64), macOS (Intel and Apple Silicon), and 64-bit Windows. Building from the source distribution requires Rust 1.85 or newer. pgen-polars currently supports Polars 1.31 through 1.42; this bounded range protects the lazy reader from incompatible changes to Polars' unstable Python IO-plugin interface.

Status: eager and lazy/chunked .bed, .pgen, and BGEN Layout 2 reading is implemented, including ordered sample/variant selection and long or samples-by-variants wide output. Statistics remain deferred. See the roadmap and product specification for details.

Layout

crates/pgen-core     Python-free Rust core (parsing, decoding, scanning)
crates/pgen-rs       `pgen-rs` command-line tool (TSV output, no Python)
crates/pgen-python   pyo3 bindings, builds the native _pgen_polars module
python/pgen_polars   Pure-Python package exposing the public API
tests/               Test data (tests/examples) and Python tests
docs/                Product spec, milestones, technical decisions

Development

Requires a Rust toolchain and uv (Python 3.12+).

# Rust checks
cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace

# Build the Python extension and run tests
uv venv --python 3.12
uv pip install -e '.[dev]'
uv run pytest tests/python -q

During development you can rebuild the native module in place with:

uv run maturin develop

Use uv run maturin develop --release when measuring reader performance; the default development build is intentionally unoptimized.

Releases

The Python package version comes from [workspace.package].version in Cargo.toml. To prepare a release, update that version and CHANGELOG.md, run the checks above, and push a matching tag such as v0.1.0. The release workflow builds and tests platform wheels, builds a source distribution, and publishes the artifacts to PyPI through Trusted Publishing.

Before the first release, add a pending Trusted Publisher on PyPI with project name pgen-polars, owner idinsmore1, repository pgen_rs, workflow release-python.yml, and environment pypi. Create the matching pypi environment in the GitHub repository. No long-lived PyPI token is required. While the project is in alpha, release notes must call out compatibility changes and downstream projects should pin an exact package version.

Command-line usage

The pgen-rs binary reads metadata files and prints them as TSV — no Python required:

cargo run -p pgen-rs -- meta pvar tests/examples/toy.pvar
cargo run -p pgen-rs -- meta psam tests/examples/toy.psam
# kinds: psam | pvar | fam | bim

It also decodes .bed and .pgen genotypes, auto-detecting sidecars:

# Wide format (default): one row per sample, one column per variant.
cargo run -p pgen-rs -- pgen tests/examples/toy.bed \
    --variant-id snp0,snp1 --sample-index 0,1

# Long format remains available explicitly for row-oriented output.
cargo run -p pgen-rs -- pgen tests/examples/toy.bed --format long \
    --variant-id snp0 --sample-index 0,1

Python API

Metadata readers return Polars DataFrames:

from pgen_polars import read_psam, read_pvar, read_fam, read_bim

variants = read_pvar("tests/examples/toy.pvar")   # CHROM, POS, ID, REF, ALT
samples = read_psam("tests/examples/toy.psam")     # IID, SEX, PHENO1

BGEN v1.2 Layout 2 uses dedicated readers. A paired Oxford .sample file is required, and an adjacent <file>.bgen.bgi index is used when present.

When a .bgi exists, numeric selections query the requested SQLite rows and RSID selections query its rsid column, then decode only those BGEN records. Without one, numeric selection scans only through its highest requested index; SNP-ID selection also uses the RSID index when the pointed BGEN record confirms that both identifiers match, otherwise it scans identifying headers. Unselected probability blocks are never decompressed.

from pgen_polars import read_bgen, scan_bgen

features = read_bgen(
    "cohort.bgen",
    ref_allele="last",
    samples=["sample_42", "sample_7"],
    variants=["variant_1", "variant_2"],
)

lazy = scan_bgen(
    "cohort.bgen",
    ref_allele="last",
    samples=["sample_42", "sample_7"],
    rsids=["rs123", "rs456"],
)

Classic .sample files use ID_2; QCTOOL v2 files use ID. BGEN does not label REF, so ref_allele="first" or "last" is mandatory. Multiallelic wide output expands to one numeric feature per ALT. Embedded BGEN identifiers are validated against classic ID_2, ID_1, or ID_1_ID_2 conventions while sample filtering and output continue to use ID_2.

Generate small 8-bit dosage, 16-bit dosage, and hard-call BGEN fixtures with:

tests/make_toy_bgen_data.sh /tmp/toy-bgen

Genotype readers (both unified for .bed and .pgen):

import polars as pl
from pgen_polars import read_pgen, scan_pgen

# A PLINK 1 .bed is a valid .pgen (storage mode 0x01). Real PGEN hard calls
# and fractional unphased biallelic dosages are supported as well.
# Sidecars (.bim/.fam or .pvar/.psam) are auto-detected from the path.
# Wide is the default: one row per sample, one column per selected variant.
features = read_pgen(
    "tests/examples/toy.bed",
    samples=["per0", "per1"],
    variants=[0, 1],  # numeric indices are the bounded random-access fast path
    n_threads=4,  # None uses available parallelism; 1 is sequential
)

# Feed the numeric matrix directly to scikit-learn.
X = features.drop("ID").to_numpy()

# Long format remains available explicitly for row-oriented workflows.
long = read_pgen(
    "tests/examples/toy.bed",
    variants=[0, 1],
    output_format="long",
)

# Lazy wide scan is also the default. Projecting variant columns avoids
# decoding other selected variants.
lazy = scan_pgen(
    "tests/examples/toy.pgen",
    samples=["per0", "per1"],
    variants=["snp0", "snp1", "snp2"],
)
selected = (
    lazy.select("ID", "snp2")
    .collect()
)

# Long lazy output remains available for row/metadata predicates.
long = (
    scan_pgen(
        "tests/examples/toy.pgen",
        variants=["snp0", "snp1", "snp2"],
        output_format="long",
    )
    .filter(pl.col("chrom") == "1")
    .select("variant_id", "sample_id", "dosage")
    .collect()
)

scan_pgen chunks long output on the variant axis (keeping all selected samples for a variant together) and wide output on the sample axis. batch_size is a positive output-row hint, so a single long-format variant can exceed it. Projection pushdown can produce long metadata-only batches without decoding genotypes; in wide output it decodes only projected variant columns, and an ID-only projection does not decode genotypes. Plain head()/slice limits are pushed into native selection before decoding: wide scans retain only the requested sample prefix, while long scans retain only variants intersecting the requested row prefix and narrow samples when the prefix ends in the first variant.

Simple predicates involving only variant metadata, long-format sample_id, or wide ID values are used to preselect source indices and are not redundantly reapplied to decoded batches. Other row-local predicates, including genotype/dosage filters, are applied to each decoded batch. Aggregate/window predicates fall back to materializing the scan before filtering so their global semantics remain correct. Constructing a scan and composing lazy expressions performs no filesystem I/O. Metadata is resolved and snapshotted when Polars first requests the schema or collects the query; dynamic wide schemas therefore resolve metadata on collect_schema(). Every collection reuses that metadata snapshot but opens a fresh genotype iterator. Create a new LazyFrame to observe sidecar changes.

Lazy scanning currently uses Polars' unstable IO-plugin interface and requires Polars 1.31 or newer. Genotype files are read through immutable memory maps. Wide real-PGEN scans decode each projected record once into a temporary, column-major mapped file and then emit bounded sample chunks from that transpose. The temporary file follows the operating system's standard temp directory configuration and is deleted when the iterator is dropped. BED wide scans decode directly from their packed mapped blocks.

read_pgen and scan_pgen accept n_threads=None to use available parallelism; pass n_threads=1 for sequential decoding. The CLI exposes the same control as pgen-rs pgen --threads N. Reproducible performance comparisons with plink2 live in docs/benchmarks.md; regenerate them with uv run python benchmarks/run_benchmarks.py --generate. For the population-scale tiny-subset case, run uv run --extra benchmark python benchmarks/run_selected_benchmarks.py /data/cohort.pgen --variants 0,35018818,70037636,105056454,140075271.

Genotype DataFrames cross the native extension boundary through Arrow C Stream capsules, avoiding per-cell Python objects. CLI TSV output uses bounded, ordered parallel formatting for large tables and a direct buffered writer for small tables; stdout itself remains serial to preserve deterministic output. Wide hard calls are stored internally as one byte per genotype while retaining the public nullable Int64 Polars schema. CLI long output is decoded and formatted in bounded chunks instead of materializing the full long table.

Selected eager wide reads use Rust Polars lazy CSV scans to project only IID/ID plus a row index from .psam/.pvar. Numeric variant selections limit the metadata prefix and load only the PGEN vblocks containing those variants. ID selections apply a Polars predicate while scanning the unindexed text sidecar. When indices and feature names are already known, pass an ordered mapping such as variants={"rs123": 42, "rs456": 9001} to skip .pvar access entirely. The Rust Polars path accepts the standard tab-delimited PLINK 2 sidecar representation; .bim/.fam retain variable-whitespace parsing.

Notes: only POS (int) and CM/QUAL (float) are typed; other columns are strings. .bim alleles are exposed as ALLELE1/ALLELE2 (no REF/ALT claim); the genotype reader imputes ref = ALLELE2, alt = ALLELE1 for .bim and uses explicit REF/ALT for .pvar.

For dosage-bearing PGEN records, the dosage column contains the explicit fractional ALT dosage. When a sample has no explicit dosage, it is inferred from its hard call. The genotype column continues to report the saved hard call, so it may be ./. while dosage is non-null. In that case is_missing is false, since usable dosage data is present. Hardcall- and dosage-phase tracks are validated and discarded by the current unphased output schema. Multiallelic dosage records are not yet supported.