gwseq_io

Python library for processing bigWig, bigBed, BAM and HiC files. Backed by a C++17 core via nanobind.

Installation

pip install gwseq-io

Requires numpy, installed automatically as a dependency.

Usage

Open bigWig, bigBed, BAM and HiC files for reading

reader = gwseq_io.open(path, mode, parallel, zoom_correction, file_buffer_size, max_file_buffer_count, index_path)

reader = gwseq_io.open("path/to/file.bigwig")
reader = gwseq_io.open("path/to/file.bam")
reader = gwseq_io.open("path/to/file.hic")

Parameters:

• mode Opening mode. May be omitted as "r" (read) by default.
• parallel Number of parallel file handles and processing threads. Use -1 for recommended (24 for reading). -1 by default.
• zoom_correction Scaling factor for automatic zoom level selection based on bin size. Only for bigWig files. 1/3 by default.
• file_buffer_size Size in bytes of each file buffer for caching file reads. Use -1 for recommended (32768 or 1048576 for URLs). -1 by default.
• max_file_buffer_count Maximum number of file buffers to keep in cache. Use -1 for recommended (128). -1 by default.
• index_path Path of the index. Only for BAM files, where it defaults to the path of the file with ".bai" appended. An index is optional, but reading entries needs one.

Attributes for bigWig and bigBed files:

• main_header General file formatting info.
• zoom_headers Zooms levels info (reduction level and location).
• auto_sql BED entries declaration (only in bigBed).
• total_summary Statistical summary of entire file values (coverage, sums and extremes).
• chr_sizes Map of chromosome IDs and their sizes.
• type Either "bigwig" or "bigbed".

Attributes for BAM files:

• header Header lines, each a dict of its "type" (the two letters after the @) and its "fields".
• chr_sizes Map of reference IDs and their sizes.
• is_indexed Whether the index was found and read. Reading entries needs it.
• index_error Why the index is absent, when it is. Empty when it loaded, and empty as well when the file simply has none.

Attributes for HiC files:

• header footer General file info.
• chr_sizes Map of chromosome IDs and their sizes.
• normalizations Available normalizations.
• units Available units.
• bin_sizes Available bin sizes.

Read bigWig and bigBed values

values = reader.read_values(chr_ids, starts, ends, centers, span, ...)

values = reader.read_values(chr_ids=["chr1", "chr1"], starts=[1000, 1100], ends=[1100, 1200])
values = reader.read_values(chr_ids=["chr1", "chr1"], starts=[1000, 1100], span=100)
values = reader.read_values(chr_ids=["chr1", "chr1"], ends=[1100, 1200], span=100)
values = reader.read_values(chr_ids=["chr1", "chr1"], centers=[1050, 1150], span=100)
values = reader.read_values(chr_ids=["chr1", "chr1"], centers=[1050, 1150], span=100, strands=["+", "-"])

Parameters:

• chr_ids starts ends centers Chromosome IDs, starts, ends and centers of the locations. Both starts ends, or one of starts ends centers with span, may be specified.
• span Reading window in bp relative to starts, ends or centers. Only one of the three may be given with it. Not by default.
• strands Strand of each location, as "+" or "-" ("." and "" count as "+"). The values of a "-" location are reversed, so that every location reads from its own start. All "+" by default.
• bin_size Reading bin size in bp. May vary in output if locations have variable spans or bin_count is specified. 1 by default.
• bin_count Output bin count. Inferred as max location span / bin size by default.
• bin_mode Method to aggregate bin values. Either "mean", "sum" or "count". "mean" by default.
• full_bin Extend locations ends to overlapping bins if true. Not by default.
• def_value Default value to use when no data overlap a bin. 0 by default.
• zoom BigWig zoom level to use. Use full data if -1, or auto-detect if -2 by taking the coarsest level whose bin size is under bin_size times zoom_correction (may be the full data). Full data by default.
• progress Function called during extraction with the extracted and the total coverage in bp. Use the default callback if true. None by default.

Returns a numpy float32 array of shape (locations, bin count).

Quantify bigWig and bigBed values

values = reader.quantify(chr_ids, starts, ends, centers, span, ...)

Parameters:

• chr_ids starts ends centers span bin_size full_bin def_value zoom progress Identical to read_values method.
• reduce Method to aggregate values over span. Either "mean", "sd", "sem", "sum", "count", "min", "max", "l1norm" or "l2norm". "mean" by default.

Returns a numpy float32 array of shape (locations).

Profile bigWig and bigBed values

values = reader.profile(chr_ids, starts, ends, centers, span, ...)

Parameters:

• chr_ids starts ends centers span strands bin_size bin_count bin_mode full_bin def_value zoom progress Identical to read_values method. A "-" location takes part in the profile reversed, as it would come out of read_values.
• reduce Method to aggregate values over locations. Either "mean", "sd", "sem", "sum", "count", "min", "max", "l1norm" or "l2norm". "mean" by default.

Returns a numpy float32 array of shape (bin count).

Iterate over all bigWig and bigBed values

iterator = reader.iter_all_values(...)

iterator = reader.iter_all_values(bin_size=10)
for values in iterator:
    ...
for (chr_id, start, end), values in zip(iterator.locs, iterator):
    ...

Parameters:

• chr_ids Only walk these chromosomes. All by default.
• bin_mode full_bin def_value zoom progress Identical to read_values method. full_bin decides whether the partial bin a chromosome ends on is walked at all.
• span Window in bp for each step, rounded up to a whole number of bins. 1 000 000 by default, a million values at the default bin size.
• bin_size Identical to read_values method, but must be a whole number of bp: the windows tile the genome on this grid. 1 by default.

Returns an iterator over successive windows, each one a numpy float32 array of shape (bins), in chromosome then coordinate order. len(iterator) gives the number of windows, and iterator.locs the region of each, so the nth array covers locs[n]:

Notes:

• A window never spans two chromosomes and no bin straddles a window boundary, so concatenating the windows of a chromosome gives exactly what read_values gives for the whole of it at the same bin size. For a bigBed the values are the pileup of its entries.

Read bigBed entries

entries = reader.read_entries(chr_ids, starts, ends, centers, span, ...)

Parameters:

• chr_ids starts ends centers span progress Identical to read_values method.
• col_count Only read this number of columns (eg, 3 for chr, start and end). Must be 0 (all) or at least 3. The columns left out are never parsed, so a narrower read is a cheaper one. All by default.

Returns a list (locations) of list of entries (dict with at least "chr", "start" and "end" keys).

Read all bigBed entries

entries = reader.read_all_entries(...)

Parameters:

• chr_ids Only extract data from these chromosomes. All by default.
• col_count Identical to read_entries method.

Returns a list of entries (as in read_entries).

Iterate over all bigBed entries

iterator = reader.iter_all_entries(...)

iterator = reader.iter_all_entries()
for entries in iterator:
    ...
for (chr_id, start, end), entries in zip(iterator.locs, iterator):
    ...

Parameters:

• chr_ids col_count progress Identical to read_all_entries method.
• span Identical to iter_all_values method, but a window is measured in bp alone, there being no bins to round it to.

Returns an iterator over successive windows, each one a list of entries (as in read_entries), in chromosome then coordinate order. len(iterator) gives the number of windows, and iterator.locs the region of each.

Notes:

• A window never spans two chromosomes, and an entry reaching over a window boundary is reported by the window it starts in. Concatenating the windows gives exactly what read_all_entries returns, in the same order.

Convert bigWig to bedGraph or WIG

reader.to_bedgraph(output_path, ...)
reader.to_wig(output_path, ...)

Parameters:

• output_path Path to output file.
• chr_ids Only extract data from these chromosomes. All by default.
• bin_size zoom progress Identical to read_values method.

Convert bigBed to BED

reader.to_bed(output_path, ...)

Parameters:

• output_path chr_ids progress Identical to to_bedgraph and to_wig methods.
• col_count Only write this number of columns (eg, 3 for chr, start and end). All by default.

Read BAM entries

entries = reader.read_entries(chr_ids, starts, ends, centers, span, ...)

Parameters:

• chr_ids starts ends centers span progress Identical to bigWig read_values method.
• filter Drop unmapped alignments, improperly paired reads, secondary and supplementary records, and anything marked as failing quality control or as a duplicate. True by default.
• parse_tags Keep the optional fields of every alignment. They are only decoded when read, so this costs one small copy per alignment. True by default.

Returns a list (locations) of list of BamEntry, in the order the locations were given. Each location gets its own full list, so two overlapping locations both report the alignments they share.

Notes:

• Needs an index, as is_indexed reports. Reading without one raises, naming either the index that was not found or the error it gave.

Read all BAM entries

entries = reader.read_all_entries(...)

Parameters:

• chr_ids Only extract data from these references. All by default.
• filter parse_tags progress Identical to read_entries method.

Returns a list of BamEntry (as in read_entries). Unplaced alignments are left out, the index reaching an alignment only through the reference it sits on.

Iterate over BAM entries

iterator = reader.iter_entries(chr_ids, starts, ends, centers, span, ...)

iterator = reader.iter_entries(chr_ids=["chr1", "chr1"], starts=[1000, 1100], ends=[1100, 1200])
for entries in iterator:
    ...
for loc_index, entries in zip(iterator.order, iterator):
    ...

Parameters:

• chr_ids starts ends centers span filter parse_tags progress Identical to read_entries method.
• sort_locations Read the locations in reference and position order, and report them in that order. Locations close together then share the blocks a read decompressed, 4 to 7 times faster on a scattered request. False by default.

Returns an iterator over locations, each one a list of BamEntry. len(iterator) gives the number of locations, and iterator.order the request index of each, so the nth list belongs to location order[n]:

Iterate over all BAM entries

iterator = reader.iter_all_entries(...)

for entries in reader.iter_all_entries():
    ...

Parameters:

• chr_ids filter parse_tags progress Identical to read_all_entries method.
• span Window in bp for each step. 1000000 by default.

Returns an iterator over successive windows, each one a list of BamEntry, in reference then coordinate order. len(iterator) gives the number of windows.

Notes:

• A window never spans two references, and an alignment reaching over a window boundary is reported by the window it starts in. Concatenating the windows gives exactly what read_all_entries returns, in the same order.

BAM entries

A BamEntry is one alignment.

Attributes:

• chr (str) Reference the alignment sits on.
• start end (int) 0-based half-open span, the end derived from the cigar. Equal for an alignment covering no reference.
• read_name (str) QNAME.
• flag (int) FLAG, as the raw bitfield.
• mapping_quality (int) MAPQ.
• cigar (str) CIGAR, eg "10S80M10S".
• sequence (str) SEQ, unpacked from its 4-bit encoding.
• qualities (str) QUAL as phred+33, "*" for a missing score.
• next_chr (str) RNEXT, "*" when the mate sits on no reference.
• next_start (int) PNEXT.
• template_length (int) TLEN.
• bai_bin (int) Index bin the record declares itself in.
• reference_length query_length (int) Bases of the reference the alignment covers, and of the read its cigar consumes.
• tags (dict) Optional fields by two-letter tag, in the order the record stores them. Typed as the file types them: character, integer, float, string, or list of integers or floats. Empty when parse_tags is off.
• is_paired is_proper_pair is_mapped is_next_mapped is_reverse is_next_reverse is_first_in_pair is_last_in_pair is_secondary_or_supplementary is_failed_qc_or_duplicate (bool) The flag bits, decoded. Finer ones are yours to mask off flag.

Notes:

• cigar, sequence, qualities and tags are decoded the first time they are read and kept afterwards, so an alignment read for its coordinates never pays for the rest of it. The optional fields are only walked when tags is read, so a record with malformed ones reads fine and raises there.
• to_dict() returns every field as a plain dict under the same names, for pandas, for serialising, or for sending to another process, an alignment itself not being picklable. It holds a "tags" key only when parse_tags was on.

Read HiC values

values = reader.read_values(chr_ids, starts, ends, ...)

Parameters:

• chr_ids starts ends Chromosome IDs, starts and ends of the two locations.
• bin_size Input bin size or -1 to use the smallest. Must be available in the file. Smallest by default.
• bin_count Approximate output bin count. Takes precedence over bin_size if specified by selecting the closest bin size resulting in bin_count. Not specified by default.
• exact_bin_count Resize output to match bin_count (if specified). Not by default.
• full_bin Extend locations ends to overlapping bins if true. Not by default.
• def_value Default value to use when no data overlap a bin. 0 by default.
• triangle Skip symmetrical data if true. Not by default.
• min_distance max_distance Min and max distance in bp from diagonal for contacts to be reported. All by default.
• normalization Either "none" or any normalization available in the file, such as "kr", "vc" or "vc_sqrt". "none" by default.
• mode Either "observed" or "oe" (observed/expected). "observed" by default.
• unit Either "bp" or "frag". "bp" by default.
• save_to Save output to this .npz path (under "values" key) and return nothing. Not by default.

Returns a numpy float32 array of shape (loc 1 bins, loc 2 bins).

Read HiC sparse values

values = reader.read_sparse_values(chr_ids, starts, ends, ...)

Parameters:

• chr_ids starts ends bin_size bin_count exact_bin_count full_bin def_value triangle min_distance max_distance normalization mode unit save_to Identical to read_values method.

Returns a COO sparse matrix as a dict with keys:

• values Values as a numpy float32 array.
• row Values rows indices as a numpy uint32 array.
• col Values columns indices as a numpy uint32 array.
• shape Shape of the dense array as a tuple.

Convert in python using scipy.sparse.csr_array((x["values"], (x["row"], x["col"])), shape=x["shape"]).

Open bigWig and bigBed files for writing

writer = gwseq_io.open(path, mode, type, chr_sizes, genome, fields, items_per_slot, compression_level)

with gwseq_io.open("path/to/file.bigwig", "w", genome="mm10") as writer:
    ...

Parameters:

• mode Opening mode. Must be set to "w" (write).
• type Type of file to write. Either "bigwig" or "bigbed". "bigwig" by default.
• chr_sizes Map of chromosome IDs and their sizes. Every written coordinate is checked against it, and a written ID is resolved against its keys the way a read one is, so "1" and "chr1" reach the same entry. Inferred from what is written if omitted, a chromosome then ending where its last value or entry does. None by default.
• genome Genome ID to get the chromosome IDs and their sizes, as get_chr_sizes returns them. May not be set with chr_sizes. None by default.
• fields Entries keys and their types. Only for bigBed files. The first three are the coordinates whatever they are called, and go out as the standard chrom, chromStart and chromEnd. Types may be "string", "int", "uint" or "float". {"chr": "string", "start": "uint", "end": "uint", "name": "string"} by default.
• items_per_slot Values or entries one block holds, and records one zoom block holds. Use -1 for recommended (1024 for bigWig, 512 for bigBed, as the UCSC writers use). -1 by default.
• compression_level zlib level for the data and zoom blocks, or 0 to leave them uncompressed. 6 by default.
• parallel Number of threads compressing blocks. Deflate is nearly all of what writing a block costs, and blocks are compressed independently, so this is where the writing time goes. 1 compresses on the calling thread and starts no threads at all. Use -1 for recommended (one per core). -1 by default.

Attributes:

• path type closed Path being written, "bigwig" or "bigbed", and whether close has run.
• chr_sizes Chromosome sizes as they will be written, in the order the chromosomes were written.
• section_count section_counts Sections, or blocks of entries, written so far — in total and, for a bigWig, by encoding ("bedgraph", "varstep", "fixedstep"). Each section takes whichever of the three costs the fewest bytes.
• entry_count fields Entries written so far, and the columns they are written with (bigBed only).
• skipped_count Values dropped for not being finite.
• clipped_count Values cut back to the end of a declared chromosome they hung over.

Notes:

• Only chromosomes that were actually written go into the file, so chr_sizes and genome are a bounds check and a spelling of the names rather than a list of what the file will contain.
• A bigWig value that starts inside a declared chromosome and ends past it is written up to that end rather than refused, since a chromosome is rarely a whole number of bins long. One that starts at or past the end raises, as does any bigBed entry running past it.

Write bigWig values

writer.write_value(chr_id, start, end, value)
writer.write_values(chr_id, start, span, values)

writer.write_value("chr1", start=1000, end=1010, value=0.1)
writer.write_values("chr1", start=1000, span=10, values=[0.1, 0.3, 0.2, 0.1])

Parameters (write_value):

• chr_id start end Chromosome ID, start and end of value.
• value Location value.

Parameters (write_values):

• chr_id start Chromosome ID and start of first value.
• span Window in bp of each successive locations relative to their starts, so values[n] covers [start + n * span, start + (n + 1) * span).
• values Locations values, as a list or a numpy array. A C-contiguous float32 array is read where it stands; anything else is converted, which copies.

Notes:

• Values must be pooled by chromosome, added in order and without overlap.
• For better performances, sequential calls should be successive locations with identical spans. A run handed over in one write_values call is a fixedStep section, four bytes a value against twelve for the same values written one at a time.
• A NaN or infinite value is not written. It leaves a gap, which is what a bigWig means by a base carrying no data, and a reader fills it with the def_value it was asked for. skipped_count counts them.
• A value hanging over the end of a declared chromosome is written up to that end, which is what makes a span that does not divide a chromosome ordinary rather than an error. clipped_count counts them. A value starting at or past the end raises, and in a write_values run only the last value can hang over, so a run reaching whole values past the end raises too.

Write bigBed entries

writer.write_entry(chr_id, start, end, ...)

writer.write_entry("chr1", start=1000, end=1010, fields={"name": "read#1"})

Parameters:

• chr_id start end Chromosome ID, start and end of entry.
• fields Map of additional fields as specified in file fields. The first three declared fields are the coordinates and are written from start and end, so naming one here is an error. A declared field left out is written empty for a string and 0 for a number.

Notes:

• Entries must be pooled by chromosome and added in order of their start. Unlike bigWig values, they may overlap and nest freely.
• The summary statistics a bigBed carries, and its zoom levels, describe the depth of coverage its entries make, as the format asks: a base under three entries counts once towards the bases covered and three towards the sum.

Convert bedGraph or WIG to bigWig

gwseq_io.convert_to_bigwig(input_path, output_path, ...)

gwseq_io.convert_to_bigwig("track.bedgraph.gz", "track.bigwig", genome="mm10")

Parameters:

• input_path Path to input bedGraph or WIG file. May be gzipped.
• output_path Path to output file.
• bin_size Force a specified bin size in the output. A bin holds the base-weighted mean of what falls in it, so a 500 bp interval counts for five hundred times what a 1 bp one does, and a bin nothing covers is left as a gap. Takes bins as is by default.
• chr_sizes genome items_per_slot compression_level parallel Identical to open in write mode.
• progress Function called during conversion. Takes the bytes read and the total size of the input as parameters. Use default callback function if true. None by default.

Returns a map of format ("bedgraph" or "wig", as it was sniffed), line_count, item_count, skipped_count, clipped_count (values cut back to the end of their chromosome) and chr_sizes as written.

Notes:

• Which of the two formats the input is comes from its content, not from its name: the first line that is neither blank, a comment, nor a track or browser declaration decides. A fixedStep or variableStep line makes it a WIG, four columns of chromosome, start, end and value a bedGraph, and anything else is refused. The format is settled once, so a file holding both is refused as well.
• WIG coordinates are 1-based and bedGraph ones 0-based half-open. step and span both default to 1; a declaration with no chrom, or a fixedStep with no start, is an error rather than a guess.
• Values must be pooled by chromosome, in order and without overlap, as write_value asks — what sort -k1,1 -k2,2n gives. Input that is not raises, naming the line. Nothing is sorted or spooled, so a conversion of any size holds a megabyte of input and one open section.
• Nothing on disk is a bigWig until the call returns, exactly as for a writer.

Convert BED to bigBed

gwseq_io.convert_to_bigbed(input_path, output_path, ...)

gwseq_io.convert_to_bigbed("peaks.bed.gz", "peaks.bigbed", genome="mm10")

Parameters:

• input_path Path to input BED file. May be gzipped.
• output_path Path to output file.
• chr_sizes genome items_per_slot compression_level progress Identical to convert_to_bigwig.
• fields Entries keys and their types, as in open in write mode. Taken from the standard BED columns — chrom, chromStart, chromEnd, name, score, strand, thickStart, thickEnd, itemRgb, blockCount, blockSizes, blockStarts, then field13 and up — for however many columns the first record has, by default.

Returns a map as convert_to_bigwig does, with format always "bed".

Notes:

• Lines are split on tabs, a BED being tab-delimited and its name column being allowed to hold spaces. Every record must carry the same number of columns as the first one, a bigBed storing one shape of record.
• A BED carries no column names of its own, so a file whose columns are named otherwise needs fields to keep them.
• Entries must be pooled by chromosome and in order of their start, but may overlap and nest freely, as write_entry allows.

Convert SAM to BAM

Not implemented yet.

gwseq_io.convert_to_bam(input_path, output_path)

Parameters:

• input_path Path to input SAM file. May be gzipped.
• output_path Path to output file.

Get genome chromosome sizes

gwseq_io.get_chr_sizes(genome, ...)

Parameters:

• genome Genome name (eg, "mm10").
• full Include unplaced chromosomes if true. Not by default.

Returns a map of chromosome IDs and their sizes, sorted by chromosome ID. Genomes that are not bundled, and any call with full, are fetched from api.genome.ucsc.edu and cached for the process lifetime.

Dev notes

Project layout

gwseq_io/
├── CMakeLists.txt          # CMake build (nanobind module)
├── pyproject.toml          # PEP 517 build config (scikit-build-core)
├── docs/                   # Files formats specifications
└── src/
    ├── cpp/
    │   ├── binding/        # nanobind bindings, one module per format
    │   ├── genomes.cpp     # Built-in genome chromosome sizes
    │   ├── bbi/            # bigWig / bigBed reader, writer and text converters
    │   ├── hic/            # HiC reader
    │   ├── bam/            # BAM reader (header, BAI index, records)
    │   └── util/           # C++17 utility library (see util/README.md)
    └── gwseq_io/
        └── __init__.py     # Python package entry-point

Every module under src/cpp is a self-contained .cpp guarded by #pragma once and #included by binding/binding.cpp, so only binding/binding.cpp is compiled — see src/cpp/util/README.md.

binding/ holds one module per format — bbi.cpp, bam.cpp and hic.cpp, each registering its own types through a bind_*(m) function — plus util.cpp for the conversions they share and binding.cpp for the module itself, open() and the other free functions.

Build from source

nanobind and scikit-build-core are resolved by pip from pyproject.toml, so they never need to be installed by hand, and CMake and Ninja are added the same way when the system does not already provide a suitable version. What has to come from the OS is the compiler toolchain, the Python headers, and — to avoid a from-source build of the dependencies — the curl and zlib development packages.

A curl built from source uses the platform's native TLS backend on Windows (Schannel) and macOS (Secure Transport), but OpenSSL on Linux, which is why the OpenSSL headers are listed in the Linux commands below.

Prerequisites — Windows

Install Visual Studio Build Tools 2019 or newer and tick the Desktop development with C++ workload — it brings MSVC, the Windows SDK, CMake and Ninja. Then install Python (the official installer ships the headers and libs) and Git for Windows.

With winget:

winget install Microsoft.VisualStudio.2022.BuildTools --override "--wait --passive --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended"
winget install Python.Python.3.12
winget install Git.Git

Windows has no system curl or zlib to link against, so both are cloned and built on the first configure — expect a noticeably longer initial build. Run the build from a Developer Command Prompt (or Developer PowerShell) so that MSVC is on the path.

Prerequisites — macOS

The Command Line Tools provide clang, git, and the system curl and zlib:

xcode-select --install

That is enough on its own, since pip fetches CMake and Ninja. To use the system CMake and a Python other than the one shipped with macOS:

brew install cmake ninja python

Prerequisites — Linux

# Fedora / RHEL
sudo dnf install gcc-c++ cmake ninja-build git \
                 python3-devel libcurl-devel zlib-devel openssl-devel

# Debian / Ubuntu
sudo apt install build-essential cmake ninja-build git \
                 python3-dev python3-venv libcurl4-openssl-dev zlib1g-dev libssl-dev

# Arch
sudo pacman -S --needed base-devel cmake ninja git python curl zlib openssl

Arch ships headers with the runtime packages, so curl, zlib and openssl cover both.

Build

# 1. Create and activate a virtual environment (recommended)
python -m venv .env
source .env/bin/activate       # Windows: .env\Scripts\activate

# 2. Build and install the package in editable mode, with the dev extras
pip install -e ".[dev]"

To build a wheel instead of installing in place:

pip install build
python -m build --wheel