Stream Crystal Reports XML at memory bandwidth.

crxml

High-performance Crystal Reports XML → Arrow/DataFrame engine for Python.

Parse, filter, rename, cast, and project Crystal Reports XML directly into
columnar data, with Rust execution, parallel parsing, bounded-memory
processing, and automatic query fusion.

Quick start

from crxml import CrystalXMLSource

source = CrystalXMLSource("report.xml", row_tag="Details")

# Row iteration: yields dicts lazily
for row in source:
    print(row["invoice"], row["amount"])

# DataFrame (auto-routes to parallel engine)
df = source.to_dataframe()
print(df.head())

That is it. df is a pandas DataFrame with zero-copy ArrowDtype strings, built in under a second for a 100 MB file.

For maximum throughput, declare the schema upfront:

schema = ["Level","Section","Field22","Field23","Field38","Field39",
          "Field61","Field73","FieldG","Text20"]
src = CrystalXMLSource("report.xml", row_tag="Details", schema=schema)
batches = src.iter_record_batches(memory="64MB", threads=16)

With pipeline stages fused into the Rust parse loop:

from crxml.stages import RenameFields, DropFields

pipeline = source | RenameFields({"f1": "invoice"}) | DropFields(["temp_id"])
df = pipeline.to_dataframe()

Performance tip: When your column set is known, pass schema=[...] to skip column discovery and enable the fast path. On production data this is the single largest performance lever (crxml goes from 4.2 GB/s to 7.6 GB/s on a 533 MB report), and the row_satisfied projection skip reaches 11 GB/s on benchmarks.

Why crxml

This library was originally inspired by carlosplanchon/xmlstreamer.

Crystal Reports XML exports are deeply nested: <Group> wraps <GroupHeader> wraps <Section> wraps <Details> wraps <Field>/<Text>/<FormattedValue>/ <Value>/<TextValue>. Standard XML libraries (ElementTree, SAX, lxml) spend most of their CPU time descending into children you do not need.

crxml skips the nesting:

• The stream engine walks the XML once with a hand-rolled memchr scanner (src/crxml_core/src/xml/scanner.rs, scan_one_row scanner.rs:119 via RowSink src/crxml_core/src/lib.rs:603) and yields flat dicts: 508 MB/s 100 MB.
• The parallel engine memory-maps the file, splits it at row boundaries (splitter.rs:57 find_split_points), and parses each chunk on its own thread into Arrow buffers directly (no dicts): up to 4.2 GB/s on high-cardinality production reports (533 MB real par128 4231) and 4.2 GB/s on uniform exports (1 GB par128 4158) via rypipe (rypipe-core Vec<ColumnBuilder>+field_index engine.rs:16, row_dirty engine.rs:26).
• Pipeline stages that rename, cast, drop, or filter fields execute in the Rust parse loop, before any Python object is created.

Comparison: stream vs parallel vs parallel streaming (bounded, schema)

Auto discovery (16x2 MiB windows for >128 MB) adds ~15% (19 ms on 533 MB) so auto is -10% vs par128 (3828 vs 4231) but still bounded and incremental. Explicit schema=[...] (FrozenSchema::from_plan) avoids Discovery and is +80% vs par128 (7630 vs 4231). Fastest bounded mode needs explicit schema; auto is safe and bounded but slightly slower. Use iter_record_batches(memory="64MB", threads=16, schema=[...]) for the fast path.

Full benchmark details: like-for-like Table vs Vec, chunk-per-cell, fixed-chunk isolation, and schema cost.

Install

pip install crxml

The columnar and parallel engines are included by default. For performance profiling counters: pip install -e . --config-settings=--features=profile.

Features

Engine guide: parallel streaming (explicit schema) is opt-in

Pass engine= explicitly, or let auto select per call. auto stays "parallel if it fits" (blocked: auto discovery adds 15% and would make auto slower until cheaper). Streaming is opt-in via iter_record_batches(..., threads=16), keeping 4 MB for par (src/crxml/source.py:164), 2 MB via budget/(threads*2) for streaming. Provide schema= for the fast path.

# Recommended bounded paths
from crxml import CrystalXMLSource
import pyarrow as pa, pyarrow.parquet as pq

src = CrystalXMLSource("report.xml", row_tag="Details")
# explicit schema: fastest, no Discovery, writer succeeds
schema = ["Level","Section","Field22","Field23","Field38","Field39","Field61","Field73","FieldG","Text20"]
src = CrystalXMLSource("report.xml", row_tag="Details", schema=schema)
batches = src.iter_record_batches(memory="64MB", threads=16)
# auto: stable but pays 15% Discovery (16×2 MiB windows for >128 MB)
batches = src.iter_record_batches(memory="64MB", threads=16)

# ParquetWriter (now works; batches share schema)
it = src.iter_record_batches(memory="64MB", threads=16)
first = next(it)
w = pq.ParquetWriter("out.parquet", first.schema)
w.write_batch(first)
for b in it: w.write_batch(b)
w.close()

Framework support

Limitations

• UTF-8 input only. UTF-16 exports (which Crystal Reports can produce) fail validation; convert first.
• No compressed input. .gz/.zst files must be decompressed before parsing.
• Crystal Reports grammar, not general XML. The flat-row model fits CR exports; arbitrary XML documents are out of scope.
• Linux-tuned performance. madvise hints and thread-count ratios were measured on Linux; other platforms work but are untested territory.
• No async API. Row iteration is synchronous.

Documentation

Full docs at crxml.emiliano-go.com covering:

• All CrystalXMLSource parameters
• Pipeline stages and fusion rules
• Sink reference
• Batch iteration and parallel distribution
• Performance with phase breakdowns
• Architecture and correctness

License

MIT