Vortex-RDF for Python

Python bindings for Vortex-RDF, a columnar RDF store format built on Vortex. Stores are opened memory-mapped from .vortex files and queried in place, without loading the dataset into memory. The bindings are read-only, with no in-memory build: build .vortex files with serialize_rdf (file → file), then open and query them.

A separate vortex-rdflib package builds an rdflib integration on these bindings; see its own documentation.

Install

pip install vortex-rdf

Quick start

from vortex_rdf import VortexRdfStore, serialize_rdf

serialize_rdf("data.nt", "data.vortex", layout="dictionary")   # RDF file -> .vortex file
store = VortexRdfStore("data.vortex")                            # lazy open; layout auto-detected
store.count_quads(p="<http://xmlns.com/foaf/0.1/name>")          # match count, no terms materialized
store.get_quads(p="<http://xmlns.com/foaf/0.1/name>")            # [(s, p, o, g), ...]

Reading quads

Every read takes a pattern as the keyword arguments s, p, o, g; an omitted position is a wildcard. Terms cross the boundary as N-Triples strings (<iri>, _:b0, "lit"@en, "3"^^<http://www.w3.org/2001/XMLSchema#integer>); the graph of a quad in the default graph is the empty string, which is also how a pattern selects it. A malformed term raises ValueError; a failing store operation raises VortexRdfError. That includes opening a store written by vortex-rdf 0.11 or earlier: its root layout is refused, and the error says to rebuild it from its RDF source with serialize_rdf.

len(store)                                                   # number of quads
store.layout()                                               # "dictionary" | "default" | "typed-object"
store.indexes()                                              # e.g. ["secondary-by-reference"]
store.count_quads(p="<http://xmlns.com/foaf/0.1/name>")      # int
store.get_quads(p="<http://xmlns.com/foaf/0.1/name>")        # [(s, p, o, g), ...]
store.match_columns(p="<http://xmlns.com/foaf/0.1/name>")    # (subjects, predicates, objects, graphs)

get_quads returns whole quads; match_columns returns the same rows transposed into four parallel columns, for callers that work a position at a time. Both are served from the term-code columns whenever the store can (Dictionary layout) and from the matched quads otherwise; results are identical. On the code path a term that repeats down a column is usually one shared Python string: a small cache of recently decoded codes serves the repeats, and a code it has dropped is decoded again as an equal, separate string.

Term codes (low-level)

For Dictionary-layout stores, match_codes returns the matched rows as four zero-copy u64 term-code columns (U64Column) — memoryview(col).cast("Q") views the Rust memory directly — decodable through a term_dict() handle. A dictionary holds at most 231 - 1 terms; a code is a Python int from 0 to 264 - 1, and anything else is refused (OverflowError or ValueError), never narrowed:

cols = store.match_codes(p="<http://xmlns.com/foaf/0.1/name>")  # (s, p, o, g) or None
dictionary = store.term_dict()                                    # TermDict or None
subjects = memoryview(cols[0]).cast("Q")
dictionary.decode(subjects[0])                       # N-Triples string for that code
dictionary.decode_many(cols[0])                      # bulk-decode a whole column
dictionary.encode("<http://xmlns.com/foaf/0.1/name>")  # code for a term, or None

decode_many decodes a batch in one GIL-released call. Buffer-protocol inputs — a column straight from match_codes, memoryview(col).cast("Q"), an array("Q", ...), a uint64 or non-negative int64 NumPy array — are read in a single bulk copy with no per-element int conversion; a raw-byte buffer whose length is a multiple of 8 (memoryview(col) itself, before the cast) is read as native-endian u64s; a buffer of any other items (a cast("I") view, a uint32 array) raises ValueError rather than being read element by element. Any sequence of ints works too. encode is the inverse of decode and tolerant of spelling: an IRI with or without angle brackets, a literal with an explicit xsd:string type or an upper-case language tag, and the default graph as "", default or [] all resolve to the stored form's code (a malformed term raises ValueError); encode_many does a batch. Both term_dict() and match_codes return None when the code path does not apply (a non-Dictionary layout). A file store's dictionary is read from the mapped file on demand — TermDict.file_backed says so — with the same calls.

Consumers can join, count, and de-duplicate entirely in code space and decode each distinct term once, never materializing a term string for a row they discard. The handle and the columns carry the pieces a query layer pushes below a pattern:

NAME = "<http://xmlns.com/foaf/0.1/name>"
d = store.term_dict()
lo, hi = d.prefix_range("<http://xmlns.com/foaf/0.1/")   # codes of one namespace: one range
literals = d.prefix_range('"')                            # codes of every literal

# A FILTER over the candidates a query produced: sorted, unique codes.
cols = store.match_codes(p=NAME)
codes = sorted(memoryview(cols[2].distinct()).cast("Q"))
passed, undecided = d.filter_codes("num_lt", "42", codes)
passed, undecided = d.filter_codes("regex", "^ali", codes, flags="i")
passed, undecided = d.filter_codes("contains", '"bob"', codes, case="lower", as_str=True)

# Narrow inside the store, before a row is gathered: a code set or range per position.
cols = store.match_codes(p=NAME, keep={"o": range(lo, hi)})
cols = store.match_codes(p=NAME, keep={"o": passed, "g": [0]}, limit=100, offset=20)
n = store.count_quads(p=NAME, keep={"o": passed}, limit=1)   # an existence test

# Many probes in one GIL-released call, answered in input order.
views = store.match_codes_many([(None, NAME, None, None), {"s": "<http://ex.org/bob>", "limit": 5}])
counts = store.count_quads_many([{"p": NAME, "keep": {"o": literals}}])

# Column kernels, order-preserving: first-seen distinct values, nested-loop joins.
s, p, o, g = cols
s.distinct(); o.value_counts()
left_idx, right_idx = o.join_indices(s)     # rows where o == s, as index pairs
o.take(left_idx)                            # gather a joined column

filter_codes(kind, arg, codes, *, flags="", case=None, as_str=False) evaluates a FILTER predicate over the candidate codes (sorted and unique, else ValueError) and answers (passed, undecided), both subsets of codes; a candidate in neither fails, and the undecided ones are left to the caller's own evaluator. No verdict is memoized. Kinds: is_literal, is_iri, is_blank; datatype <iri>, lang <tag>, lang_matches <range>; num_lt/num_le/num_gt/num_ge/num_eq/num_ne <number> (value comparison for well-formed numeric literals; xsd:long/xsd:unsignedLong beyond 64 bits undecided; different XSD datatypes order by their IRIs; a non-literal fails the orderings and =, passes !=); and the string kinds str_prefix <p>, contains/strstarts/strends <constant's N-Triples spelling>, regex <pattern> with flags. String kinds follow rdflib 7.6: the text is a string literal's lexical form (with as_str=True, SPARQL STR(): an IRI's string or a literal's lexical form; a blank node, or a literal whose datatype rdflib normalizes, is undecided), case="lower"|"upper" wraps that text (Python's str.lower()/str.upper(), decided on ASCII text only), a language-tagged constant needs the same tag on the text, and regex decides only patterns from an allow-listed subset that agrees with Python's re; the stub's docstring has the exact rules.

keep takes a dict from position ("s", "p", "o", "g" or 0–3) to a code set or a code range. A set is a U64Column, a u64 buffer or a list of ints. A range is a range with step 1 (a stepped range raises ValueError) or a 2-tuple (lo, hi), the half-open codes lo <= code < hi. A 2-tuple is always a range, so a set of codes is passed as a list or a U64Column, never as tuple(codes). limit/offset window the rows in base order, and a filtered file scan stops at the first block that fills the window. A probe of the *_many calls is an (s, p, o, g) tuple or a dict with keys s, p, o, g, keep, limit, offset; every probe is parsed before any is evaluated.

Build options

serialize_rdf(input_path, output_path, *, format=None, layout="dictionary", indexes=[])

serialize_rdf writes beside output_path and renames the finished file into place, so a build that fails (a parse error halfway through the input, say) leaves no partial file and an existing store untouched. A store file the process cannot write (a read-only file) is never replaced: serialize_rdf raises PermissionError. On Linux and macOS rebuilding a path that an open VortexRdfStore has mapped is safe: the open store keeps reading the file it mapped. On Windows the rename is refused while a store has the file mapped, so the rebuild fails with an OSError until that store, and every TermDict and U64Column taken from it, is dropped. Every option after the two paths is keyword-only. format is an RDF format name ("ntriples", "nquads", "turtle", "trig", "n3", "rdfxml", "jsonld", or the short aliases nt, nq, ttl, rdf, xml), detected from the input file extension when omitted. Opening auto-detects the layout and indexes — VortexRdfStore takes no layout argument; store.layout() and store.indexes() report the same names.

layout — how terms are encoded into columns. "dictionary" is the default in every vortex-rdf frontend (Python, JS and the CLI):

indexes — secondary access paths, each costing extra space:

Bytes & files

A file store is memory-mapped: opening reads only the footer (and, under the Dictionary layout, the dictionary's window bounds), and every query reads the pages it touches straight from the file. What stays in RAM is the operating system's page cache, which shows up as file-backed RSS (RssFile) and is reclaimable, plus at most 4 MiB of rebuilt dictionary windows per memory-mapped store, which every VortexRdfStore(path) is — the process's own (anonymous) memory stays small whatever the store's size. The mapping lives as long as the store, or any TermDict or U64Column taken from it, does. Never truncate or rewrite a store file in place while it is open (serialize_rdf replaces it by rename instead); network filesystems are not supported. VortexRdfStore(path, in_memory=True) loads the whole store into memory instead, so each subsequent match skips the file.

Stores also round-trip through bytes: store.to_bytes() serializes to the native container (the same exchange format as the .vortex file, the CLI and the JS bindings), and VortexRdfStore.from_bytes(data) opens such a buffer — bytes or bytearray — as a fully in-memory store.

The bindings start their worker threads on first use. A process that forks after calling into them (opening a store, serialize_rdf) leaves the child without them, and batch calls in the child hang. Call into the bindings only after forking, or use the spawn or forkserver start method.

Development

Managed with uv; maturin runs under the hood as the build backend:

cd python
uv sync                      # creates .venv, builds + installs the extension
uv run pytest tests          # run the test suite
uv run maturin develop --uv  # fast rebuild while iterating on Rust code

Rust source changes are picked up by uv sync automatically (see [tool.uv] cache-keys in pyproject.toml). Without uv: python -m venv .venv && pip install maturin pytest "rdflib>=7.6,<8" && maturin develop && pytest tests (rdflib is the oracle of tests/test_filter_codes_differential.py).

Building from source (the sdist or a development build) additionally requires libclang: a transitive build dependency of the Vortex file engine (custom-labels, via vortex-io) generates C bindings with bindgen at compile time. It is preinstalled on most dev setups (Xcode, LLVM on Windows); on Linux install e.g. clang-devel (dnf) or libclang-dev (apt). Installing a published wheel needs none of this.

Benchmarks

bench/run.py measures these bindings against pyoxigraph, pycottas, rdflib and lightrdf on a file → store → query workload and writes bench/results.json for the dashboard's Python tab; bench/test_codspeed.py is the instrumented suite CodSpeed runs.

python3 python/bench/run.py                 # full run
BENCH_DIM=32 python3 python/bench/run.py    # quick pilot
uv run pytest bench/test_codspeed.py --codspeed

Harness design (per-library virtualenvs, dataset parity with js/bench/datasets.ts, unsupported cells where a library lacks the operation, matched-row counts cross-checked and any disagreement recorded in config.countWarnings) is documented in bench/run.py, bench/worker.py and bench/adapters.py. Configuration variables:

License

MIT