super-xml

Distributed, splittable XML record parsing built on the succinct DOM — packaged as a PySpark custom DataSource with no JVM native code, so it runs on Databricks Serverless where JNI/JAR-native data sources are blocked.

import superxml

COLUMNS = [
    ("title", "text", ["title"], "", "utf8"),
    ("id",    "attr", [],        "id", "utf8"),
]

# Split a large file by byte range across executors (default).
df = superxml.read(spark, "/Volumes/.../wiki_dump.xml", "page", COLUMNS)

# Same call, one flag, for a directory of many small files.
df = superxml.read(spark, "/Volumes/.../manyfiles/", "order", COLUMNS,
                   many_small_files=True)

df.count()  # triggers the distributed parse

Why this exists

The succinct DOM parses a real 5.24 GB XML file (DBLP) in ~45 s at 0.75× the file size in memory — beating lxml, roxmltree, and ElementTree on both memory and speed, and the only one of them that parsed that file at all. super-xml makes that engine usable from Spark for files too big for one machine, on serverless runtimes that forbid JVM-native data sources.

The design in one paragraph

Spark hands each task a byte range of a huge file. parse_partition realigns the range to whole-record boundaries (standard Hadoop XmlInputFormat ownership: a partition owns records whose opening tag begins in its range), wraps the owned records in a synthetic root, and feeds that to the unmodified hardened core parser. So every bit of the core's correctness work (namespaces, entities, UTF-8 validation, fuzzing) carries over, and splitting is provably lossless — see the exhaustive "cut at every byte offset" parity test in src/lib.rs.

Distribution rides on Spark's Python DataSource API (Spark 4.0+ / DBR 15.4+ / serverless env v2): partitions() plans byte ranges on the driver, read() calls the Rust core (via PyO3, GIL released) on executors. Pure Python + a CPython native extension — no JVM .so, so serverless allows it.

Install

pip install super-xml        # import name is `superxml`

The distribution ships a prebuilt abi3 manylinux wheel (CPython ≥ 3.9), so no Rust toolchain is needed to install. read_many_small_files additionally needs pyspark>=4.0 and a runtime with pandas>=2.2.0 (Databricks and any real Spark cluster already provide these — see pyproject.toml for why they are not hard-pinned).

Results

All on Databricks Serverless (environment v2), input wiki_139972.xml (1,753 MB MediaWiki dump, 139,972 <page> records). Full detail: databricks/RESULTS.md.

• Lossless: 139,972 / 139,972 parsed, zero loss or duplication across the distributed byte-range partitions.
• 1.72× faster than native Spark XML on the same count() (full parse + extract), with exactly matching record counts — a fair speed comparison, not accuracy-for-speed. Native XML runs entirely in the JVM with no Python-worker hop or FFI copies; super-xml wins despite paying all of that.

One call, two paths

superxml.read(spark, path, row_tag, columns, *, many_small_files=False) is the single entry point. It dispatches to one of two underlying readers:

• default — the SuperXmlSource DataSource (format("superxml")). Splits large files by byte range across executors. Use for files too big for one machine. Extra **kwargs pass through as DataSource options.
• many_small_files=True — routes a directory of many small files through Spark's JVM-side binaryFile listing/reads (~3.2× faster than the DataSource on 65k small files). Refuses large files it cannot split, via a size guard (opt out with skip_size_stats=True). Extra **kwargs forward to it.

The two paths can't be a single format(...).option(...) call: the DataSource reader runs on an executor with no SparkSession, so it can't invoke binaryFile itself — that must happen on the driver. read hides that split behind one flag. You can still call either underlying reader directly.

Which do I use?

Rule of thumb: reach for many_small_files=True only when every file is small (comfortably under one executor's memory). For anything else — a single huge file, or a directory that mixes small and huge — use the default; it is the only path that decides per file.

Why the flag is not "auto": the fast path loads each file whole into one task (no byte-range splitting), so it cannot handle a file bigger than a task's memory. It guards against this — with many_small_files=True, a single file over max_file_bytes (default 256 MB) raises ValueError and points you at the default reader, rather than silently OOM-ing. (Setting skip_size_stats=True removes that guard and the guard's one-time size scan — only do that when you already know every file is small.) The default DataSource has no such limit: its driver-side router bin-packs the small files and byte-range-splits the huge ones in the same pass, so a mixed directory "just works".

Build & test (from source)

Building from source needs the sibling Rust crate xml-dom-compress checked out alongside this repo (it is a path dependency), plus maturin.

cargo test                        # pure-Rust core + parity suite, no Python needed
maturin build --release --features python --out dist   # build the wheel

Known v1 limitations

• Single rowTag. Heterogeneous record types (DBLP's article/inproceedings/…) need a set of tags; v1 supports one (perfect for wiki <page>). Multi-tag is a planned extension.
• UTF-8 input. ISO-8859-1 documents (like raw DBLP) need transcoding to UTF-8 first; the core rejects non-UTF-8 by policy.

License

MIT — see LICENSE.