gffbase 0.2.1

What is GFFBase?

A high-performance genomic-annotation engine — a SIMD Rust parser, a DuckDB columnar backend and a zero-copy PyArrow interface — built for whole-genome ingest and bulk machine-learning feature extraction, and a drop-in successor to gffutils.

The legacy FeatureDB / Feature / create_db / DataIterator / GFFWriter / merge_criteria API is preserved verbatim, so most scripts migrate by changing one import line.

Install

pip install gffbase

Universal abi3 wheels: one binary per platform covers CPython 3.10 through 3.14, and no Rust toolchain is needed. Source and development builds are in the installation guide.

0.2.1 is the current release — correctness fixes to 0.2.0, several of which silently lost data; upgrade. Coming from 0.1.0, 0.2.x also fixes two SQL injection vulnerabilities (advisory) and has breaking changes. Both are in the changelog.

Quick start

from gffbase import create_db

# Ingest a GTF or GFF3 — format auto-detected, gzip transparent.
with create_db("gencode.v49.basic.annotation.gtf.gz", "gencode.duckdb") as db:

    # Walk one gene's hierarchy. GENCODE ids carry their version.
    for tx in db.children("ENSG00000139618.19", level=1, featuretype="transcript"):
        print(tx.id, tx.start, tx.end)

    # Overlap query, routed to the per-seqid R-tree.
    for exon in db.region("chr17:43044295-43125483", featuretype="exon"):
        print(exon)

Coming from gffutils? import gffbase as gffutils is usually the only change you need — but read the migration guide first, for the one loop pattern you should not port unchanged.

Bulk extraction without Python objects

The workload GFFBase exists for: pull every exon for tens of thousands of transcripts, hand the columns to a tensor, train. One set-based query returns DuckDB's own Arrow buffers, and no Feature object is constructed at any layer — where a per-feature Python loop allocates millions of throwaway objects and that allocation dominates everything else.

exons = db.children_batched(
    transcript_ids,              # an iterable of 50 000 IDs
    featuretype="exon",
    format="arrow",              # "df" and "polars" also supported
)
# A pyarrow.Table sharing memory with DuckDB. Its "anchor" column carries the
# input id for each row, so per-transcript groups survive without N queries.

import torch
starts = torch.from_numpy(exons.column("start").to_numpy())

region_batched() and parents_batched() offer the same zero-copy contract for spatial and parent workloads. One trade to know: iterating for i in ids: db.children(i) is slower here than legacy's row-by-row SQLite path, because DuckDB pays vectorization startup on every call — which is why the batched API exists.

End-to-end PyTorch and Hugging Face pipelines are in the ML workflows cookbook, and every method has a snippet in the usage gallery.

Measured against gffutils

Head-to-head across five canonical human-genome annotations — one run, one machine, one commit. Read the ingest column as a draw rather than a win: gffbase spans 1.21× to 0.69×, ahead where per-feature overhead dominates and behind on the attribute-dense whole-genome files, because both engines are attribute-bound and effectively serial at a comparable rate. The durable advantages — batched extraction, spatial indexing, SQL over the whole corpus — are untouched by that result.

Measured on AMD EPYC 7702 64-Core Processor · 128 cores · 1007.22 GB RAM · Linux-5.14.0-503.15.1.el9_5.x86_64-x86_64-with-glibc2.34
Versions: Python 3.11.16 · gffbase 0.2.0rc1 · duckdb 1.5.5 · pyarrow 25.0.1 · gffutils 0.14
Commit: 42bb900e328c · Run: 2026-09-10T19:56:52Z
Generated from benchmarks/results/06_mega.linux-x86_64.json by tools/gen_benchmark_tables.py. Do not edit by hand.

peak RSS is ingest plus exhaustive validation, not the cost of ingest: validate_db defaults to sample=200 and the CLI never overrides it. No speedup is published unless both engines' correctness signatures match first, and every number is generated from a committed measurement file — a test fails if a published table stops matching it. The per-corpus analysis, the fairness constraints and the measured run-to-run spread are on the performance and methodology pages.

What's inside

• Rust + PyO3 parser — SIMD line and tab splitting, percent-decoding in and encoding back out, GTF semicolon-in-quotes safe, gzip transparent, failures raised as line-numbered GFFFormatError.
• DuckDB columnar storage — an 11-table schema plus 3 compatibility views, set-based GTF gene/transcript synthesis, recursive-CTE transitive closure, and a per-seqid-banded R-tree built inline during ingest.
• Smart routing — region() picks R-tree or B-tree; children() picks the closure cache or a dynamic CTE from the corpus's measured hierarchy depth.
• Vectorized batched API — children_batched, parents_batched and region_batched return pyarrow.Table, pandas.DataFrame or polars.DataFrame straight out of DuckDB's buffer pool.
• Drop-in legacy API — plus bed12, interfeatures, an execute() SQL escape hatch and export_sqlite(), with every remaining difference from gffutils 0.14 declared and tested.
• Discontinuous features — several lines sharing one ID are one logical feature, with per-segment phase, covered_length and explode_segments=.
• A command line and validation — gffbase create|fetch|children|parents|region|search|rmdups|sanitize|validate|migrate, structural validation at two depths, and in-place migration of a v1 database.

Documentation

Full site: khchao.com/gffbase

Contributing

Pull requests, bug reports and feature suggestions are welcome. CONTRIBUTING.md covers the Rust and Python development setup, the test suite and its coverage gates, branch naming and the PR checklist. The repository ships issue and pull-request templates.

License

Apache License 2.0 — see LICENSE.

Citation

If GFFBase contributes to your research, please cite it:

@software{chao_gffbase_2026,
  author  = {Chao, Kuan-Hao},
  title   = {{GFFBase}: Rust-accelerated GFF3/GTF parser with a
             DuckDB-backed storage engine and zero-copy PyArrow interface},
  year    = 2026,
  version = {0.2.1},
  url     = {https://github.com/Kuanhao-Chao/gffbase},
}

The repository also ships a CITATION.cff, so GitHub's "Cite this repository" button produces an up-to-date reference. Per-version DOIs are tracked on the releases page.