gffbase 0.3.0

What is GFFBase?

A high-performance genomic-annotation engine — a SIMD Rust parser, a DuckDB columnar backend and a zero-copy PyArrow interface — built for whole-genome ingest and bulk machine-learning feature extraction, and a drop-in successor to gffutils.

The legacy FeatureDB / Feature / create_db / DataIterator / GFFWriter / merge_criteria API is preserved verbatim, so most scripts migrate by changing one import line.

Install

pip install gffbase

Universal abi3 wheels: one binary per platform covers CPython 3.10 through 3.14, and no Rust toolchain is needed. Source and development builds are in the installation guide.

0.3.0 is the current release — ingest 2.4–2.9× faster, in under a third of the memory on whole-genome files; loops over a live stream prefetched; and noisy files read as they were meant. Coming from 0.2.0, upgrade: 0.2.1 fixed defects that silently lost data. Coming from 0.1.0, 0.2 and later fix two SQL injection vulnerabilities (advisory) and have breaking changes. All are in the changelog.

Quick start

from gffbase import create_db

# Ingest a GTF or GFF3 — format auto-detected, gzip transparent.
with create_db("gencode.v49.basic.annotation.gtf.gz", "gencode.duckdb") as db:

    # Walk one gene's hierarchy. GENCODE ids carry their version.
    for tx in db.children("ENSG00000139618.19", level=1, featuretype="transcript"):
        print(tx.id, tx.start, tx.end)

    # Overlap query, routed to the per-seqid R-tree.
    for exon in db.region("chr17:43044295-43125483", featuretype="exon"):
        print(exon)

Coming from gffutils? import gffbase as gffutils is usually the only change you need — but read the migration guide first, for the one loop pattern you should not port unchanged.

Bulk extraction without Python objects

The workload GFFBase exists for: pull every exon for tens of thousands of transcripts, hand the columns to a tensor, train. One set-based query returns DuckDB's own Arrow buffers, and no Feature object is constructed at any layer — where a per-feature Python loop allocates millions of throwaway objects and that allocation dominates everything else.

exons = db.children_batched(
    transcript_ids,              # an iterable of 50 000 IDs
    featuretype="exon",
    format="arrow",              # "df" and "polars" also supported
)
# A pyarrow.Table sharing memory with DuckDB. Its "anchor" column carries the
# input id for each row, so per-transcript groups survive without N queries.

import torch
starts = torch.from_numpy(exons.column("start").to_numpy())

region_batched() and parents_batched() offer the same zero-copy contract for spatial and parent workloads. One trade to know: a loop over a gffbase iterator (for g in db.features_of_type("gene"): db.children(g)) is prefetched and runs within 1.4–3.5× of gffutils, but a loop over a list of ids of your own, for i in ids: db.children(i), is slower — each call is a DuckDB statement of its own — which is why the batched API exists.

End-to-end PyTorch and Hugging Face pipelines are in the ML workflows cookbook, and every method has a snippet in the usage gallery.

Measured against gffutils

Head-to-head across five canonical human-genome annotations — one run, one machine, one commit. gffbase ingests every corpus faster, 1.92× to 3.62×, into a database 0.61× to 0.89× the size of gffutils' SQLite file. The narrowest margin is the GENCODE GTF row, the arm least favourable to gffbase: with parent inference off, gffutils' GTF path is a plain bulk insert. Batched extraction, spatial indexing and SQL over the whole corpus come on top.

Measured on AMD EPYC 7702 64-Core Processor · 128 cores · 1007.22 GB RAM · Linux-5.14.0-503.15.1.el9_5.x86_64-x86_64-with-glibc2.34
Versions: Python 3.11.16 · gffbase 0.3.0 · duckdb 1.5.5 · pyarrow 25.0.1 · gffutils 0.14
Commit: 632a4d80dee0 · Run: 2026-09-27T15:07:53Z
Generated from benchmarks/results/06_mega.linux-x86_64.json by tools/gen_benchmark_tables.py. Do not edit by hand.

peak RSS is ingest plus exhaustive validation, not the cost of ingest: validate_db defaults to sample=200 and the CLI never overrides it. No speedup is published unless both engines' correctness signatures match first, and every number is generated from a committed measurement file — a test fails if a published table stops matching it. The per-corpus analysis, the fairness constraints and the measured run-to-run spread are on the performance and methodology pages.

What's inside

• Rust + PyO3 parser — streaming, SIMD line and tab splitting, percent-decoding in and encoding back out, GTF semicolon-in-quotes safe, gzip and one-file tar archives transparent, failures raised as line-numbered GFFFormatError.
• Rust ingest producer — ids, duplicates and batching without a Python object per record, straight into DuckDB as Arrow; memory bounded by a budget.
• DuckDB columnar storage — an 11-table schema plus 3 compatibility views, set-based GTF gene/transcript synthesis, a materialized transitive closure, and a per-seqid-banded R-tree built inline during ingest.
• Smart routing — region() picks the R-tree or a zone-map-pruned scan; children() and parents() look ids up in one table, and inside a loop over a gffbase iterator are prefetched.
• Noisy input — byte-order marks, CR line endings, CDS-only and AUGUSTUS GTF, Liftoff ids, truncated gzip: read as meant, and reported (noisy files).
• Vectorized batched API — children_batched, parents_batched and region_batched return pyarrow.Table, pandas.DataFrame or polars.DataFrame straight out of DuckDB's buffer pool.
• Drop-in legacy API — plus bed12, interfeatures, an execute() SQL escape hatch and export_sqlite(), with every remaining difference from gffutils 0.14 declared and tested.
• Discontinuous features — several lines sharing one ID are one logical feature, with per-segment phase, covered_length and explode_segments=.
• A command line and validation — gffbase create|fetch|children|parents|region|search|rmdups|sanitize|validate|migrate, structural validation at two depths, and in-place migration of a v1 database.

Documentation

Full site: khchao.com/gffbase

Contributing

Pull requests, bug reports and feature suggestions are welcome. CONTRIBUTING.md covers the Rust and Python development setup, the test suite and its coverage gates, branch naming and the PR checklist. The repository ships issue and pull-request templates.

License

Apache License 2.0 — see LICENSE.

Citation

If GFFBase contributes to your research, please cite it:

@software{chao_gffbase_2026,
  author  = {Chao, Kuan-Hao},
  title   = {{GFFBase}: Rust-accelerated GFF3/GTF parser with a
             DuckDB-backed storage engine and zero-copy PyArrow interface},
  year    = 2026,
  version = {0.3.0},
  url     = {https://github.com/Kuanhao-Chao/gffbase},
}

The repository also ships a CITATION.cff, so GitHub's "Cite this repository" button produces an up-to-date reference. Per-version DOIs, when available, are tracked on the releases page.