sigilyx

YXDB reader and writer for Python.

YXDB is the native binary format used by Alteryx Designer. sigilyx reads and writes .yxdb files using Polars DataFrames, PyArrow Tables, or Pandas DataFrames.

The core is written in Rust. No native Alteryx Designer installation is required.

Format scope: sigilyx has full read/write support for the E1 (original engine) YXDB layout. Experimental read support for E2 (AMP engine) is included - 15 field types have been verified against real E2 files; the rare Time and WString types retain speculative decoders behind an opt-in flag. E2 writing is not yet supported. E2 records that cannot be decoded raise an error rather than silently becoming null rows. See SPECIFICATION-E2.md for details.

Installation

pip install sigilyx                     # Polars only (default)
pip install "sigilyx[arrow]"            # + PyArrow
pip install "sigilyx[pandas]"           # + Pandas + PyArrow
pip install "sigilyx[all]"              # all extras

Requires Python 3.9+. Pre-built wheels for Windows, macOS, and Linux (x64 and ARM).

Quick Start

import polars as pl
import sigilyx  # importing registers pl.read_yxdb(), df.yxdb, etc.

# Read
df = pl.read_yxdb("data.yxdb")

# Write
df.yxdb.write("output.yxdb")

API

Polars Integration

Importing sigilyx registers official Polars namespace plugins and top-level IO aliases. No extra calls needed - just import sigilyx.

import polars as pl
import sigilyx

# Top-level IO (mirrors pl.read_parquet / pl.scan_parquet style)
df = pl.read_yxdb("data.yxdb")           # returns pl.DataFrame
lf = pl.scan_yxdb("data.yxdb")           # returns pl.LazyFrame (IO plugin)

# Namespace API on DataFrame / LazyFrame
df.yxdb.write("output.yxdb")             # pl.DataFrame → .yxdb file
lf.yxdb.sink("output.yxdb")              # pl.LazyFrame → .yxdb file (streaming)

Reading

import sigilyx as yx

# Polars DataFrame - fastest, zero-copy via Arrow C Data Interface
df = yx.read_yxdb("data.yxdb")

# PyArrow Table
table = yx.read_yxdb_arrow("data.yxdb")

# Pandas DataFrame (via PyArrow)
pdf = yx.read_yxdb_pandas("data.yxdb")

Writing

import sigilyx as yx

# Polars DataFrame
yx.write_yxdb("output.yxdb", df)

# PyArrow Table
yx.write_yxdb_arrow("output.yxdb", table)

# Pandas DataFrame
yx.write_yxdb_pandas("output.yxdb", pdf)

Streaming / Batched Read

Iterate over large files with constant memory usage:

import sigilyx as yx

# Basic iteration - each batch is a Polars DataFrame
for batch in yx.read_yxdb_batches("data.yxdb", batch_size=100_000):
    process(batch)

# Column projection - only materialise the columns you need
for batch in yx.read_yxdb_batches("data.yxdb", columns=["Id", "Name", "Amount"]):
    process(batch)

# Row limit - stop after N total rows
for batch in yx.read_yxdb_batches("data.yxdb", n_rows=5_000):
    process(batch)

Lazy Scan

import polars as pl
import sigilyx as yx

# Returns a Polars LazyFrame backed by a native Rust streaming reader.
# Only the YXDB header is read on construction; data streams on .collect().
lf = yx.scan("data.yxdb")

result = lf.filter(pl.col("amount") > 100).collect()

# Projection pushdown - only selected columns are materialised in Rust
top10 = lf.select("id", "name").head(10).collect()

Pushdown support: projection (select / with_columns) and row-limit (n_rows / .head()) are pushed down to the Rust reader. Predicate pushdown is not possible - YXDB rows are LZF-compressed with no block-level statistics, so .filter() is applied after the scan.

Metadata

import sigilyx as yx

# Inspect schema without reading any row data
fields = yx.read_yxdb_fields("data.yxdb")
for f in fields:
    print(f.name, f.field_type, f.size)

# Record count from the file header - no data read
n = yx.record_count("data.yxdb")

Diagnostics

Create a privacy-aware report when a file does not read:

python -m sigilyx.diagnostics data.yxdb

The default report includes environment, file size and SHA-256, detected format, schema type counts, record count, and decode outcome. It excludes the input path, file name, row values, raw bytes, schema names, and full parser errors. Use --include-schema or --include-error-details only after reviewing the output for sensitive information.

Field Types

Links

• GitHub: https://github.com/Sigilweaver/SigilYX
• Documentation: https://sigilweaver.app/sigilyx/
• Rust crate (crates.io): https://crates.io/crates/sigilyx
• Changelog: https://github.com/Sigilweaver/SigilYX/blob/main/CHANGELOG.md
• Issues: https://github.com/Sigilweaver/SigilYX/issues

License

Apache License 2.0.