genoray

If you want to use NumPy with genetic variant data, genoray is for you! genoray enables ergonomic and efficient range queries of genotypes and dosages from VCF and PGEN (PLINK 2.0) files. genoray is also fully type-safe and has minimal dependencies.

Summary

The genoray API more-or-less boils down to just two classes and up to five methods:

  • VCF and PGEN classes for reading VCF and PGEN files, respectively.
  • read read variants for a single range.
  • chunk read variants for a single range in chunks.
  • read_ranges read multiple ranges of variants at once.
  • chunk_ranges read multiple ranges of variants in chunks.
  • set_samples subset and/or re-order the samples.

The other important arguments to know are mode (and phasing for VCF) to set the return type and max_mem for chunking. The modes that are available for each file format are always accessible from the class itself, e.g. VCF.Genos16, PGEN.GenosDosages, etc. You can also filter variants on the fly using the filter argument to class constructors.

Also included:

  • SparseVar and SparseVar2 sparse variant stores for compact, range-queryable on-disk representations of genotype data.
  • Reference for reading reference genome sequence.
  • Mutation catalogues and signature refitting via cosmic_signatures and fit_signatures.
  • A genoray index|write|view CLI for building indices, converting VCF/PGEN to sparse formats, and inspecting variant files.

See the genoray-api skill and the docs for details.

Examples

VCF

We work with VCFs using the (you guessed it) VCF class:

from genoray import VCF

vcf = VCF("file.vcf.gz")

Querying data for a region is as simple as:

# shape: (samples ploidy variants)
genos = vcf.read("1")  # read all variants on chromosome 1

You can also change the return type to be either genotypes and/or dosages by providing a mode argument:

vcf = VCF("file.vcf.gz", dosage_field="DS")  # need a dosage_field to read dosages

genos, dosages = vcf.read("1", mode=VCF.Genos16Dosages)

Dosages have shape (samples, variants) and dtype np.float32.

[!NOTE] VCFs must also be provided a FORMAT dosage_field to read dosages and this field must have Number=A in the header, meaning there is one value for each ALT allele.

A key feature of genoray is letting you work with data that is too large to fit into memory. For example:

vcf = VCF("file.vcf.gz", phasing=True)  # include phasing status

# max_mem defaults to "4g", can also be capitalized or be "GB", for example
# Genos8 reduces precision to int8 from the default int16 that cyvcf2 uses
genos = vcf.chunk("1", max_mem="4g", mode=VCF.Genos8)

for chunk in genos:
    # do something with chunk, each chunk is a NumPy array of shape (samples, ploidy+1, variants)
    ...

The chunk method will automatically chunk the data along the variants axis to respect the memory limit, returning a generator of data instead of everything at once.

[!NOTE] We also set phasing=True and changed the mode to VCF.Genos8 to read phased genotypes as int8. The phasing argument lets us have access to the phase of the genotype in the format the cyvcf2 adheres to: the 3rd entry along the ploidy axis is the phase: 0 for unphased, 1 for phased. Reducing precision to int8 instead of int16 reduces the memory per variant by half -- we would only need higher precision if we expected to have more than 128 alleles at a variant site.

PGEN

from genoray import PGEN

pgen = PGEN("file.pgen")

[!IMPORTANT] PGEN files are automatically indexed on construction, creating a <prefix>.gvi file. This is a one-time cost to enable fast range queries, but it takes longer for larger files. Don't delete this index file unless you want to re-index the PGEN file.

We can query data for a region in the same way as VCF:

# shape: (samples ploidy variants)
genos = pgen.read("1")  # read all variants on chromosome 1
genos = pgen.chunk("1")  # read all variants on chromosome 1

However, PGEN files also support reading multiple ranges at once since this improves throughput substantially:

# shape: (samples, ploidy, variants), shape: (n_ranges+1)
genos, offsets = pgen.read_ranges('1', starts=[1, 1000, 2000], ends=[1000, 2000, 3000])
first_range_genos = genos[..., offsets[0]:offsets[1]]

genos = pgen.chunk_ranges('1', starts=[1, 1000, 2000], ends=[1000, 2000, 3000])
for range_ in genos:
    if range_ is None:
        # no data for this range
        continue
    for chunk in range_:
        # do something with chunk, each chunk is a NumPy array of shape (samples, ploidy, variants)
        ...

The read_ranges method takes starts and ends and returns data for each range and the offsets to slice out the variants for each range. Since the data is allocated as a single array, the offsets let you slice out the data for each range from the variants axis.

[!NOTE] We do not provide an API for multi-range queries of VCFs because benchmarking showed that this provided no benefit to throughput.

Like VCF, methods for PGENs accept a mode argument to change the return type to include genotypes, phasing, and/or dosages:

genos, phasing, dosages = pgen.read("1", mode=PGEN.GenosPhasingDosages)

The PGEN reader adheres to pgenlib's API, so the phasing information is in a separate boolean array instead of using an extra column like VCF/cyvcf2. The phasing information is a boolean array of shape (samples, variants) where True indicates that the genotype is phased and False indicates that it is unphased.

[!IMPORTANT] PGEN files either store hardcalls (genotypes) or dosages, not both, and dosage PGENs infer hardcalls based on a hardcall threshold. Thus, if you want to read hardcalls that do not correspond to inferred hardcalls from a dosage PGEN, you can provide two different PGEN files to the constructor. This will read hardcalls from hardcalls.pgen and dosages from dosage.pgen. The two PGEN files must have the same samples and variants in the same order. The dosage_path argument is optional, and if not provided, both hardcalls and dosages will be sourced from the path argument ("hardcalls.pgen" in the example):

pgen = PGEN("hardcalls.pgen", dosage_path="dosage.pgen", ...)

Filtering

You can filter variants from VCF or PGEN files by a providing a function or polars expression to the constructor, respectively.

For VCFs, the function must accept a cyvcf2.Variant and return a boolean indicating whether to include the site.

# only include variants that are common in EUR
vcf = VCF("file.vcf.gz", filter=lambda v: v.INFO['AF_EUR'] > 0.05)

For PGENs, the expression operates on the .gvi index — a polars DataFrame with columns:

  • CHROM — contig name
  • POS — 1-based position
  • REF — reference allele
  • ALT — list of alternate alleles
  • ILEN — list of indel lengths (one per ALT: len(ALT) - len(REF), or a signed size for symbolic SVs; null for un-sizable symbolic/breakend alleles)

Prefer the ready-made expressions in genoray.exprs — is_snp, is_indel, is_biallelic, is_symbolic, is_breakend, is_imprecise, and ILEN — and combine them with polars operators. For custom predicates, use pl.col("CHROM"/"POS"/"REF"/"ALT"/"ILEN") directly.

import genoray
from genoray import PGEN

# only include SNPs
pgen = PGEN("file.pgen", filter=genoray.exprs.is_snp)

# exclude symbolic alleles and breakends
pgen = PGEN("file.pgen", filter=~genoray.exprs.is_symbolic & ~genoray.exprs.is_breakend)

⚠️ Important ⚠️

  • For the time being, ploidy is 2 for all classes in genoray, but this could be more flexible for VCFs in the future. The PGEN format does not support ploidy other than 2.
  • Different file formats may use different data types for their respective representations of genotypes, phasing, and dosages.
  • Ranges are 0-based, so starts begin at 0 and ends are exclusive.
  • Missing genotypes and dosages are encoded as -1 and np.nan, respectively.
  • Dosages from PGEN files may not exactly match VCF files (up to a fraction of a percent) because PLINK 2.0 must encode dosages with fixed precision which can not match what can be represented by text in a VCF (may also disagree with how BCF encodes dosage).

Contributing

To contribute to genoray, please fork the repository and create a pull request. We welcome contributions of all kinds, including bug fixes, new features, and documentation improvements. Please make sure to run the tests before submitting a pull request. We provide a Pixi environment that includes all development dependencies. To use the environment, install Pixi and run pixi run prek-install to activate pre-commit in your clone of the repo, and then run pixi s in the repository root directory. pixi s will activate the development environment and install all dependencies. You can then run the tests using pytest. ❗Note that all commits must adhere to conventional commits. If you have any questions or suggestions, please open an issue on the repository.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (4.0.1)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for dlaub from gravatar.com dlaub

Credits

Author: David Laub

Requires

Python <3.15, >=3.10

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

genoray-3.3.1.tar.gz (2.1 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

genoray-3.3.1-cp310-abi3-manylinux_2_28_x86_64.whl (2.5 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.28+ x86-64

genoray-3.3.1-cp310-abi3-manylinux_2_28_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.28+ ARM64

genoray-3.3.1-cp310-abi3-macosx_11_0_arm64.whl (2.2 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

File details

Details for the file genoray-3.3.1.tar.gz.

File metadata

  • Download URL: genoray-3.3.1.tar.gz
  • Upload date:
  • Size: 2.1 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: uv/0.12.0 {"installer":{"name":"uv","version":"0.12.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for genoray-3.3.1.tar.gz
Algorithm Hash digest
SHA256 4442d758b8db13b1cd3d3ea490711d710ae440c0ff019a2bfb2af53fb1d36c4e
MD5 696612b70b75a6c40f155c89580f18b0
BLAKE2b-256 ddd2eacca57c6ecb16bf73f98d6b011c801889ea12850b0f59cbcd6c5d61aac9

See more details on using hashes here.

File details

Details for the file genoray-3.3.1-cp310-abi3-manylinux_2_28_x86_64.whl.

File metadata

  • Download URL: genoray-3.3.1-cp310-abi3-manylinux_2_28_x86_64.whl
  • Upload date:
  • Size: 2.5 MB
  • Tags: CPython 3.10+, manylinux: glibc 2.28+ x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: uv/0.12.0 {"installer":{"name":"uv","version":"0.12.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for genoray-3.3.1-cp310-abi3-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 fb7c314e013835db6bef9ab6a940219908c24888a4dd574239f1987381022ed1
MD5 198b9df6cca0f469f06b341adc6b3fe6
BLAKE2b-256 8402a6253af89cf532e8029c7508423848704e8362d9e3080bdc5e27f34b7b12

See more details on using hashes here.

File details

Details for the file genoray-3.3.1-cp310-abi3-manylinux_2_28_aarch64.whl.

File metadata

  • Download URL: genoray-3.3.1-cp310-abi3-manylinux_2_28_aarch64.whl
  • Upload date:
  • Size: 2.6 MB
  • Tags: CPython 3.10+, manylinux: glibc 2.28+ ARM64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: uv/0.12.0 {"installer":{"name":"uv","version":"0.12.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for genoray-3.3.1-cp310-abi3-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 de27b6cf4461f30ab3b3a050cd6a41c3a83cbd36378315e73639051baa398861
MD5 d25792f4f67d1a1b3d68991ffd9c3d91
BLAKE2b-256 955c1222a9cf39999f336068c11ff640d4ed5a3f1dbc2e2b739039c5ffd9b7ca

See more details on using hashes here.

File details

Details for the file genoray-3.3.1-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

  • Download URL: genoray-3.3.1-cp310-abi3-macosx_11_0_arm64.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: CPython 3.10+, macOS 11.0+ ARM64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: uv/0.12.0 {"installer":{"name":"uv","version":"0.12.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for genoray-3.3.1-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 95c013e328b3deee12a7a5cd2aeb340469ca0f12f668cbcbe1740aca94229473
MD5 e32a945aad57c6c2798c02a6a69560c1
BLAKE2b-256 94070884ea45f06ace0f1ee423d9184f40d598303ba746ff85a6a9a35b56cb94

See more details on using hashes here.

Release history Release notifications | RSS feed

4.0.1

4 files

4.0.0

4 files

3.4.0

4 files

This release

3.3.1 This release

4 files

3.3.0

4 files

3.2.1

4 files

3.2.0

4 files

3.1.0

4 files

3.0.0

4 files

2.15.0

2 files

2.14.0

2 files

2.13.0

2 files

2.12.3

2 files

2.12.2

2 files

2.12.1

2 files

2.12.0

2 files

2.11.1

2 files

2.11.0

2 files

2.10.0

2 files

2.9.2

2 files

2.9.1

2 files

2.9.0

2 files

2.8.0

2 files

2.7.3

2 files

2.7.2

2 files

2.7.1

2 files

2.7.0

2 files

2.6.0

2 files

2.5.0

2 files

2.4.0

2 files

2.3.3

2 files

2.3.2

2 files

2.3.1

2 files

2.3.0

2 files

2.2.3

2 files

2.2.2

2 files

2.2.1

2 files

2.2.0

2 files

2.1.3

2 files

2.1.2

2 files

2.1.1

2 files

2.1.0

2 files

2.0.1

2 files

2.0.0

2 files

1.0.1

2 files

1.0.0

2 files

0.17.0

2 files

0.16.1

2 files

0.16.0

2 files

0.15.0

2 files

0.14.6

2 files

0.14.5

2 files

0.14.4

2 files

0.14.3

2 files

0.14.2

2 files

0.14.1

2 files

0.14.0

2 files

0.13.1

2 files

0.13.0

2 files

0.12.2

2 files

0.12.1

2 files

0.12.0

2 files

0.11.3

2 files

0.11.2

2 files

0.11.1

2 files

0.11.0

2 files

0.10.8

2 files

0.10.7

2 files

0.10.6

2 files

0.10.5

2 files

0.10.4

2 files

0.10.3

2 files

0.10.2

2 files

0.10.1

2 files

0.10.0

2 files

0.9.0

2 files

0.8.0

2 files

0.7.1

2 files

0.7.0

2 files

0.6.0

2 files

0.5.1

2 files

0.5.0

2 files

0.4.4

2 files

0.4.3

2 files

0.4.2

2 files

0.4.1

2 files

0.4.0

2 files

0.3.0

2 files

0.2.0

2 files

0.1.0

2 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page