polars-pdb 0.1.0

A locally-mirrored, SQL-queryable catalogue of the PDB, sourced from PDBe, that advances one wwPDB release per week

polars_pdb

A locally-mirrored, SQL-queryable catalogue of the PDB, sourced from PDBe, that advances one wwPDB release per week.

polars-pdb update

is the whole idea: run it weekly (a cron job, ignored otherwise) and the local archive tracks upstream, one release at a time. Everything else in this project exists in service of that one command.

Status: alpha. The schema and CLI are still settling, and the mirror is a real commitment -- see Setup before you start.

Two layers, deliberately separate

Layer	Content	Role
Mirror	.cif.gz, byte-identical to PDBe	Source of truth. Full coordinates. Not read at query time.
Catalogue	Parquet: metadata, chains, sequences, cross-references	The queryable index. Derived. Rebuildable. No coordinates.

The workflow is always the same direction:

Query the catalogue in SQL or Polars → get back pdb_ids and filesystem paths → open those CIF files yourself.

No coordinates ever live in Parquet, at any scale. The coordinates are already sitting on disk in the mirror, in a format every structural biology tool already reads. Duplicating them into a second, bespoke representation would be the MMTF mistake all over again -- a clever re-modelling nobody but this project owns, and nobody but this project can fix when it goes stale. Dropping them is also what keeps the catalogue two orders of magnitude smaller than the mirror: a coordinate table at this archive's scale would be on the order of 2.6 billion rows and ~28 GB; the actual catalogue is under 1 GB.

So, explicitly, this library does not:

• load coordinates, in any form -- there is no atom table and never will be
• compute derived geometry -- no contacts, no SASA, no centre-of-mass, no neighbour search
• talk to the PDBe REST API -- it works from bulk flat files; 256k round-trips is not an ingest strategy
• know anything about assemblies beyond their definition -- an assembly is a recipe (symmetry operators, stored as text), never materialised coordinates

It has one job -- turn "what's in the PDB" into a fast SQL/Polars question -- and is judged on that job alone.

The numbers

All measured against a real sync, not estimated:

Entries	256,448
Mirror size	89.79 GB (.cif.gz, byte-identical to PDBe)
Catalogue size	well under 1 GB (the entire xref_* cross-reference family is ~50 MB)
Largest table	xref_go -- 16,281,793 rows, ~27 MB, point query ~24 ms in DuckDB
Full rebuild	~2.2 minutes across 16 cores
Typical weekly update	seconds (~300 changed entries)

Setup

pip install polars-pdb   # or: uv add polars-pdb
export POLARS_PDB_ROOT=/data/pdb   # wherever you can spare ~90 GB
polars-pdb update

$POLARS_PDB_ROOT is required, not defaulted -- the mirror alone is a ~90 GB download, and this project would rather fail loudly than pick a location for that on your behalf. The first update on an empty root is a full rebuild (same code path, no special case), so budget the time and the disk before running it.

polars-pdb status reports the latest wwPDB release against what your local archive is synced to (entry counts included), rendered as a Rich table.

Querying

Two ways in, same catalogue underneath.

Polars, lazy over the Parquet:

import polars as pl
import polars_pdb as ppdb

cat = ppdb.catalogue()                    # lazy Polars frames over catalogue/

hits = (
    cat.entries
       .filter(pl.col("method") == "X-RAY DIFFRACTION")
       .filter(pl.col("resolution") < 2.0)
       .join(
           cat.xref_uniprot.filter(pl.col("uniprot") == "P42212"),
           on="pdb_id",
       )
       .collect()
)

paths = ppdb.paths(hits["pdb_id"])        # -> filesystem paths to .cif.gz

SQL, via DuckDB directly over the Parquet:

polars-pdb query "
    SELECT DISTINCT e.pdb_id, e.resolution, x.uniprot
    FROM entries e JOIN xref_uniprot x USING (pdb_id)
    WHERE e.resolution < 2.0 AND x.uniprot = 'P42212'
"

or the same query from Python via polars_pdb.duck.query(sql), which returns a Polars DataFrame.

Note the DISTINCT: xref_uniprot is per chain segment (a chain can map to more than one UniProt range), so one entry can yield multiple rows. That's the source's real grain being honest at query time, not a bug -- see "Cross-references" below.

ppdb.paths() is the library's boundary. It returns paths and nothing else. What you do with the coordinates -- gemmi, Biopython, whatever you already use -- is yours.

The catalogue

14 tables, one Parquet file each, all under catalogue/:

entries · entities · chains · sequences · ligands · citations · assemblies · xref_uniprot · xref_pfam · xref_cath · xref_scop · xref_go · xref_ec · xref_taxonomy

Cross-references (xref_*)

There is deliberately no single xrefs table. pdb_chain_uniprot is one row per chain segment; pdb_chain_go is one row per chain/GO term -- a single (entry, chain) pair can have hundreds of GO rows and dozens of Pfam rows. Flattening those into one row per chain would mean fabricating a cross product the source data doesn't contain. Each SIFTS flatfile gets its own table, at its own native grain, 1:1 with the source; joins across them happen explicitly at query time, which is a feature, not friction.

CLI

polars-pdb update            # advance the archive by one wwPDB release
polars-pdb update --dry-run  # report what would change; write nothing
polars-pdb rebuild           # update from an empty state -- same code path
polars-pdb status            # release, drift, entry counts
polars-pdb query "SELECT …"  # ad-hoc DuckDB, Rich-rendered
polars-pdb path 1cbs         # id -> filesystem path

polars-pdb update is designed to be the thing a weekly cron calls and otherwise ignores.

Why this exists

The niche is genuinely open -- there is no substantive PDB-as-Parquet or PDB-as-DuckDB project on PyPI. The closest comparable, PDBj Mine 2, needs a ~50 GB PostgreSQL server. This is a library and a Parquet directory: no server, no daemon, pip install and a filesystem path.

The design leans on a few hard-won lessons:

• The update is the primary operation, not an afterthought. The PDB is a living archive; a one-shot dump is stale in seven days. A full rebuild is just an update from an empty state -- one code path, exercised every week, so it can't rot.
• Schema mirrors the mmCIF dictionary, not a bespoke re-modelling. MMTF died in 2024 and took its dependent tooling with it; BinaryCIF, which stayed close to the mmCIF categories, survived.
• The mirror is truth; the catalogue is a projection. Deleting the whole catalogue and regenerating it is always safe, because no fact exists only in Parquet.

See PLAN.md for the full architecture and the measurements behind it, and BUILD.md for how it was actually built.