pyhdtkit 0.3.1

A pure-Python package to convert between RDF Turtle (.ttl) and HDT (.hdt):

• .ttl → .hdt
• .hdt → .ttl
• combine two or more .hdt files into one

No CLI — import pyhdtkit is the interface. No Rust, no native extension.

Install

pip install pyhdtkit
pip install "pyhdtkit[fast]"   # optional CRC speedup, see Performance

Dev:

pip install -e ".[dev]"

Usage

from pyhdtkit import ttl2hdt, hdt2ttl, hdtcat

ttl2hdt("graph.ttl", "graph.hdt")
hdt2ttl("graph.hdt", "graph.ttl")
hdtcat(["a.hdt", "b.hdt"], "combined.hdt")

Errors

All three functions raise ValueError for anything that goes wrong — a missing or unreadable input file, malformed Turtle, a truncated or corrupt .hdt file, or an unwritable output path. hdtcat additionally requires at least 2 input paths.

Status

All three functions are implemented: a real HDT binary reader and writer (dictionary front-coding, BitmapTriples), built from scratch — no Rust, no C extension, no wrapping an existing HDT library. rdflib handles Turtle parsing/serialization; everything HDT-specific is pure Python.

The read path (hdt2ttl) is verified against a real .hdt file produced by independent hdt-cpp tooling (tests/fixtures/snikmeta.hdt), not just against our own writer.

Performance

HDT's compactness comes from succinct bit-level structures (rank/select bitmaps, front-coded dictionaries) that are naturally suited to compiled languages. This is pure Python — it will be slower and more memory-hungry than the reference C++ (hdt-cpp) or a Rust implementation, especially at large triple counts. That's an accepted, deliberate trade-off for this package: correctness and hackability over raw speed.

Measured on this machine (benchmarks/bench.py, synthetic triples, default front-coding block size):

Roughly linear scaling.

Optional speedup

pip install "pyhdtkit[fast]"

This pulls in google-crc32c — the [fast] columns above. HDT checksums every section it writes, and a pure-Python CRC loop is ~2600x slower than a compiled one, which made it the single largest cost in the read path once everything else was tuned.

Being precise about what this is: google-crc32c wraps google/crc32c, a compiled C++ library. It ships as a prebuilt wheel for CPython 3.9–3.14 on Windows x64, macOS (Intel/ARM), and glibc Linux (x86_64/i686/aarch64), so you don't need a compiler — but there is compiled C++ running under the hood, and there is no musl wheel, so on Alpine this extra would try to build from source.

None of that touches the default install: pip install pyhdtkit pulls only rdflib (itself pure Python) and contains zero compiled code. The pure-Python CRC stays the fallback, and the test suite pins the two implementations to identical output and runs green in both modes.

Only the checksum is ever delegated — all HDT encoding and decoding is our own Python code either way.

Notes on what makes it fast

• Bit-packing streams through a small bounded buffer rather than shifting one whole-array Python integer, which would be O(n²) (binio.py's pack_lsb_bitfields/unpack_lsb_bitfields).
• Bitmaps (1 bit per entry, the largest arrays in a typical file) get a byte-at-a-time fast path instead of a per-bit loop.
• Dictionary front-coding finds shared prefixes via a single big-integer XOR instead of comparing bytes one at a time.

No numpy or other compiled-array dependency was needed; one may get added later if profiling on a real workload shows it's worth the weight.