Rugo

Thin · light · opinionated for low resource usage

Rugo is the file layer extracted from the Opteryx SQL engine. It was built to keep memory use low by pushing filters as early as possible — skipping columns you don't need and pruning row groups before any data is decoded.

Why Rugo?

Thin by design. Opinionated about what not to load.

Rugo was built as the file layer for the Opteryx SQL engine, where the constraint was simple: read as little data as possible, hold as little in memory as possible. Column projection and row-group pruning happen before any decoding. The result is a library that is fast on selective queries, tiny on disk, and carries no PyArrow, Pandas, or NumPy into your environment.

Measured on Python 3.14, Apple M-series. Import times on a cold process (first load off disk).

Serverless-first

AWS Lambda and GCP Cloud Functions bill by memory and package size. At 17 MB installed and a 5 ms cold import, Rugo fits where PyArrow's 124 MB footprint doesn't.

Tiny container images

At 16× smaller than PyArrow, Rugo meaningfully shrinks image size, speeds scale-out, and keeps you well clear of AWS Lambda's 250 MB unzipped layer limit.

Read less, go faster

Column projection and row-group pruning are first-class citizens. Skip the columns you don't need. Skip the row groups that can't match. Rugo's advantage grows with selectivity.

No surprise dependencies

The wheel bundles everything it needs — Draken, the columnar substrate, ships inside. pip install rugo is the entire dependency story.

Where PyArrow is faster: full-table scans with no filtering. Rugo does not compete on decode throughput — it competes on how little it has to decode in the first place.

Quickstart

One install, four formats.

Rugo reads and writes Parquet, CSV, JSONL and Avro. The API is the same shape across all of them: pass a path or bytes, get columnar data back.

Installation

pip install rugo

Pre-built wheels bundle Draken — there is nothing else to install. Rugo has zero runtime dependencies.

Requirements:

• Python 3.11+
• A platform with a published wheel (Linux x86-64/aarch64, macOS arm64). For other platforms, see Building from source.

Data model

Rugo speaks Draken, the bundled columnar substrate:

• A Vector is a single typed column. Call vector.to_pylist() to get a Python list of its values.
• A Morsel is a batch of rows across several columns (a chunk of a table). Call morsel.column(b"name") to get a column Vector (note the bytes key), and len(morsel) for the row count.

Readers return Morsels (Parquet) or a result dict whose columns are Vectors (CSV, JSONL). The writers consume a Morsel. A read → write round-trip:

from rugo import parquet
from rugo.csv import write_csv
from rugo.jsonl import write_jsonl

with parquet.read_parquet("planets.parquet") as reader:
    for morsel in reader:                          # one Morsel per row group
        csv_bytes    = write_csv(morsel)           # -> bytes (RFC 4180)
        jsonl_bytes = write_jsonl(morsel)          # -> bytes (one JSON object per row)
        pq_bytes     = parquet.write_parquet(morsel)   # -> bytes (ZSTD)

Command-line interface

Installing Rugo puts a rugo command on your PATH — the same reader and writer, driven from the shell. No Python required at the call site; it's the quickest way to inspect a file, convert between formats, or wire Parquet/CSV/JSONL/Avro into a shell pipeline.

rugo info space_missions.parquet          # rows, columns, size, format
rugo schema space_missions.parquet        # column names, types, nullability
rugo preview -n 5 space_missions.parquet  # first 5 rows as a table
rugo convert space_missions.parquet out.jsonl   # format is inferred from the extension

Every verb takes --json to emit machine-readable output instead of a text table, so the CLI composes with jq and friends:

rugo count --json events.parquet | jq .num_rows
rugo describe --json events.parquet | jq '.columns[] | select(.null_count > 0)'

Verbs

describe, stats, and inspect read statistics from the Parquet footer, which CSV and JSONL don't carry — pointing them at a non-Parquet file is a clean error, not a crash. merge requires identical column names, order, and types across inputs and fails loud on a mismatch rather than coercing. diff reports schema differences only (column set and types), not row-level data changes.

Verb names are stable: info, schema, diff, and convert mean what you'd expect and are safe to script against.

Parquet

rugo.parquet is the recommended surface: one symmetric module for reading and writing that accepts a filename or an in-memory buffer, streams row-group Morsels, applies predicate pushdown, and writes Morsels back to bytes.

Quick start

from rugo import parquet

# Schema-only metadata (footer parse, no column data). Path OR bytes.
meta = parquet.read_metadata("planets.parquet")
print(meta.num_rows)                      # 9
print([c.name for c in meta.schema_columns])

# Streaming read: one Morsel per row group. `columns` projects; `predicates`
# prune whole row groups via footer statistics, then filter surviving rows
# exactly — the yielded morsels contain only rows that match.
with parquet.read_parquet(
     "planets.parquet",
    columns=["id", "name"],
    predicates=[("id", ">", 4)],            # ops: = == != < <= > >= in "not in"
) as reader:
    for morsel in reader:
        print(morsel.column(b"name").to_pylist())

# Write a Draken Morsel to Parquet bytes (ZSTD by default; "none" to disable).
data = parquet.write_parquet(morsel, compression="zstd")
with open("out.parquet", "wb") as f:
    f.write(data)

# Stream several morsels to a file at constant memory — one row group per
# write_row_group() call, bytes pushed to `sink` as they're produced.
with open("out.parquet", "wb") as f:
    with parquet.open_parquet_writer(f.write) as writer:
        for batch in batches:
            writer.write_row_group(batch)

rugo.parquet API

source is a filename (str) or bytes/bytearray/memoryview. predicates is a list of (column, op, value); row groups are pruned by footer statistics (and bloom filters for equality on file sources), then surviving rows are filtered exactly.

Low-level API (rugo.parquet_reader)

Most callers should use rugo.parquet above. The low-level module is exposed for fine-grained control.

Metadata

schema_columns is a tuple of SchemaColumn(name, physical_type, logical_type, nullable). read_rowgroup_stats min/max are raw stat bytes (or None); decode with decode_value.

Decode

read_parquet(data, column_names=None, row_group_mask=None)

• data — bytes, bytearray, or memoryview holding the full Parquet file.
• column_names — list[str] to project, or None for all columns.
• row_group_mask — optional iterable, one truthy/falsy entry per row group; a falsy entry skips decoding that row group (predicate pushdown). rugo.parquet's predicates= builds this from read_rowgroup_stats.
• Returns list[Morsel] (one per decoded row group), or None on failure. On partial decode failure an individual column within a Morsel may be None.

Compatibility

Fine-grained / range decode

col_stats is the per-column stats dict for the matching row group from read_metadata.

Bloom filters

bloom_filter_maybe_contains(path, bloom_offset, bloom_length, value)   # -> bool

Evaluates a column bloom filter at the given byte offset/length for a candidate value. Bloom filter offsets and lengths are exposed in the per-column metadata returned by read_metadata.

Supported decode subset

Writing

rugo.parquet_writer (and the rugo.parquet facade) serialize a Draken Morsel to a well-formed, PyArrow-readable Parquet file.

from rugo.parquet import write_parquet, write_parquet_fast, write_parquet_with_bounds
data = write_parquet(morsel, compression="zstd")           # -> bytes
data, bounds = write_parquet_with_bounds(morsel)           # + per-column min/max
data = write_parquet_fast(morsel)                          # no bloom/dictionary/statistics/page index

Unsupported column types fail loud (no silent skip). Nested LIST/MAP/STRUCT and dictionary-encoded output are not yet implemented.

Writer identity (created_by) and versions

rugo is built into two distributions, and they stamp different text into the Parquet footer's created_by:

Both strings mean "rugo wrote this". The versions in them are not comparable: the bundled build is released as part of opteryx_core and carries that version, while rugo.__version__ always reports the standalone rugo source version regardless of which wheel is installed. So in the bundled case a file says 0.9.x while rugo.__version__ says 0.4.x, and neither is wrong.

To identify the writer from Python — including which of the two you have — read the build identity rather than the version:

import rugo
rugo.__writer_id__     # 'opteryx-rugo version 0.9.120 (build 3522)'  — EXACTLY what goes in the footer
rugo.__distribution__  # 'opteryx_core' or 'rugo'
rugo.__version__       # '0.4.38' — the rugo source version, in both cases

__writer_id__ is generated at build time from the same string that is compiled into the writer, so it cannot drift from what the files actually say.

This distinction does not affect the sorted_by trust gate. A reader trusts a file's row-group sorting_columns when created_by identifies rugo, and that check is a substring search for rugo — no version is parsed, and both spellings above pass it by construction.

Streaming writer: parquet.open_parquet_writer(sink, ...) writes the same column types/encoding/compression/statistics/bloom-filter support above, but as one row group per write_row_group(morsel) call, pushing each chunk of bytes to sink (a callable taking bytes) as it's produced — footer/statistics accumulate incrementally and are emitted on close(). Peak memory is ~one row group regardless of the total file size, independent of write_parquet's whole-morsel-in/whole-file-out shape. Every batch passed to the same writer must share the same column schema.

Limitations

• Not a full Parquet replacement reader; decode support is intentionally narrow.
• LZO, BROTLI, and legacy Hadoop-framed LZ4 (parquet codec 5) are not implemented in the decode path. A file using one raises with the codec named — it is never read as zero rows. Rewrite with ZSTD or LZ4_RAW (codec 7).
• FIXED_LEN_BYTE_ARRAY decodes only as DECIMAL (width 1..16); other FLBA uses (UUID, fixed-width hashes) raise.
• Decode logic is built around DATA_PAGE (V1); DATA_PAGE_V2 is not handled.
• Decode reads from a single data-page path per column chunk; files requiring full multi-page streaming decode may return partial or failed column results.
• Nested, list, and map-heavy files are not a primary decode target; flat primitive columns are the intended shape.
• On partial decode failure, individual columns may be returned as None.
• Metadata extraction is broad, but known edge cases remain around list/nested column naming normalisation.

Performance

Metadata reads (schema + row-group stats, no column data) are fast and comparable to PyArrow. The high-level read_parquet() path decodes (row group, column) chunks and builds Draken vectors on a shared worker pool, so projected and filtered reads of multi-row-group files use all available cores; pass threads=1 for the serial path. The emphasis is still on reading less — projection and row-group pruning.

Wide file (50 cols, 200k rows, 55 MB)

On narrow files PyArrow is faster across the board. On wide files with filtering, Rugo is 2–3×+ faster — the crossover is driven by how many columns can be skipped and how many rows are eliminated before the typed column build.

JSONL

Quick start

from rugo.jsonl import get_jsonl_schema, read_jsonl, write_jsonl

# Infer schema from sample rows
schema = get_jsonl_schema("example.jsonl", sample_size=5)
# -> {"columns": [{"name": str, "type": str, "nullable": True}, ...]}

# Read from a file path with projection and predicate pushdown
result = read_jsonl(
     "example.jsonl",
    columns=["id", "name"],
    predicates=[("status", "==", "active")],
)
if result["success"]:
    print(result["num_rows"])
    for vec in result["columns"]:           # list of Draken Vectors
        print(vec.to_pylist())

# Read from bytes input
with open("example.jsonl", "rb") as f:
    result = read_jsonl(f.read(), columns=["id"])

# Write a Morsel to JSONL bytes (one JSON object per row)
data = write_jsonl(morsel)

read_jsonl

read_jsonl(
    data,                        # file path (str) or buffer (bytes/bytearray/memoryview)
    columns=None,                # list[str] to project, or None for all
    predicates=None,             # list[(column, op, value)]; op in ==, !=, <, <=, >, >=,
                                 #   in, not in (value: list/tuple/set), is null, is not null (value: None)
    explicit_schema=None,        # provide a schema dict instead of inferring
    infer_schema=True,
    infer_sample_size=5,         # rows sampled for type inference
    parse_arrays=True,            # materialize uniform-scalar JSON arrays as ARRAY vectors
    parse_objects=True,           # materialize JSON objects as VARIANT vectors
    fail_on_error=True,
    use_threads=True,            # SIMD-accelerated parallel scan/interpret
)

Return dict:

Inferred type strings: int64, double, boolean, string, null (a column absent/null on every row), plus array and variant when parse_arrays/parse_objects materialize a column (see below). explicit_schema columns are echoed back verbatim rather than inferred; other columns appear only when infer_schema=True (it does not change how the returned Draken vectors are typed, only whether they're reported here).

parse_objects (default True): a column whose sampled value is a JSON object is returned as a DRAKEN_VARIANT vector — the same raw-JSON-text storage as string, just tagged as variant rather than decomposed into fields. Set parse_objects=False to get the old string/DRAKEN_VARCHAR behaviour instead.

parse_arrays (default True): a column whose sampled value is a JSON array is materialized as a DRAKEN_ARRAY vector when every element across every row is a uniform scalar kind (all-int/double, all-boolean, all-string, or all-null/empty — ints widen to double like the scalar-column path). A row whose value is not a JSON array (a scalar or object — a string whose text looks like an array, such as "[1,2]", is a string, not an array), malformed array text, nested containers (an array inside an array) or a genuine mix of scalar kinds (e.g. [1, "a", true]) are out of scope: the column falls back to raw JSON text (string/DRAKEN_VARCHAR, same as parse_arrays=False) and a RuntimeWarning is raised naming the column.

get_jsonl_schema

get_jsonl_schema(data, sample_size=5)
# -> {"columns": [{"name": str, "type": str, "nullable": True}, ...]}

Infers the schema from the first sample_size rows. Returns {"columns": []} on failure; does not raise.

Writing

write_jsonl(morsel) returns bytes, one JSON object per row. Value formatting is done in C++: doubles use shortest round-trip (std::to_chars); dates/timestamps render ISO-8601 strings; decimals are JSON numbers; arrays render as JSON arrays (null list / empty list / null element are all distinguished); nulls are null.

Performance

116 MB, 1.5 M rows, 5 cols, versus PyArrow read_json (multithreaded):

Bulk SELECT * is materialiser-bound — PyArrow has an edge. The analytical shapes — project + filter — are 1.2–5×+ faster, and the advantage grows with selectivity and table width.

Caveats

• String/object-heavy fields are often returned as bytes (binary-preserving), not eagerly decoded Python str/dict values.
• Mixed or deeply nested array-object content may fall back to raw JSON text/bytes in edge cases.
• Schema inference's type hint is capped at infer_sample_size leading rows (the first non-null value in that window), and every row is still validated against the chosen hint (falling back to VARCHAR on a mismatch) — so a small sample never corrupts or loses a value. It CAN change the reported/stored type, though: if the sample window is all-null (e.g. a column that's null for its first N rows then numeric), no hint ever forms and the column is typed VARCHAR even where a larger sample would have picked int64/double/boolean. explicit_schema skips inference for named columns and enforces the declared type strictly: a value that doesn't fit raises ValueError rather than silently falling back. The vocabulary covers everything inference can produce, including VARIANT (a JSON object or array per row, as text) and ARRAY<T> for T in INT64/UINT64/FLOAT64/BOOL/VARCHAR, so a schema taken from one chunk can be pinned onto every other chunk of the same file; those two are JSONL-only and read_csv refuses them.
• fail_on_error=True catches a malformed line that never opens a record, a {/key abandoned by an unexpected newline, and a truncated/unterminated array or object — raising ValueError with the line number, byte offset, and a snippet. It is not a full JSON validator: some malformed shapes (e.g. a value truncated at end-of-file with no trailing newline) still parse leniently either way.

CSV

Quick start

from rugo.csv import read_csv, write_csv

result = read_csv("data.csv")                                           # all columns
result = read_csv("data.csv", columns=["col1", "col2"])                 # projection
result = read_csv("data.csv", columns=["name"], predicates=[("age", ">", 30)])
result = read_csv("data.tsv", delimiter="\t")                           # TSV variant

if result["success"]:
    for vec in result["columns"]:           # list of Draken Vectors
        print(vec.to_pylist())

# Write a Morsel to CSV bytes (RFC 4180)
data = write_csv(morsel, delimiter=",", header=True)

read_csv

read_csv(
    data,                # file path (str) or buffer (bytes/bytearray/memoryview)
    columns=None,        # list[str] to project, or None for all
    predicates=None,     # list[(column, op, value)]; op in ==, !=, <, <=, >, >=
    delimiter=",",       # field separator character
    has_header=True,     # whether the first row is a header
    use_threads=True,    # parallel scan
)

Return dict:

Type inference cascade per field: int64 → float64 → VARCHAR → null (empty field).

Writing

write_csv(morsel, delimiter=",", header=True, for_excel=False) returns RFC 4180 bytes: fields are quoted when they contain the delimiter/quote/newline (quotes doubled), nulls are empty fields, and ARRAY columns render as a (quoted) JSON array. The CSV and JSONL writers share the same C++ value formatter.

for_excel=True checks the morsel against the limits of the Excel grid the file is destined for and raises ValueError rather than writing something Excel would silently mangle (it truncates the over-long cell and drops the off-sheet rows and columns without reporting either):

The row count is per-morsel; a caller concatenating several morsels into one file has to add up the rows itself. A CSV file has no limits of its own, so this is off by default and does not otherwise change the output.

Performance

Measured against pyarrow.csv.read_csv. The expensive step is typed column build; Rugo makes it survivor-only, which pays off when there is something to skip.

Narrow file — 3 cols, 1 M rows, 12.6 MB:

Wide file — 50 cols, 200 k rows, 55 MB:

On narrow files PyArrow is faster across the board. On wide files with filtering, Rugo is 2–3×+ faster — the crossover is driven by how many columns can be skipped and how many rows are eliminated before the typed column build.

Known limitations

• Field length is capped at 65,535 bytes (uint16_t index); longer fields are silently truncated.
• Type inference is speculative from sampled values; there is no schema-override parameter — inferred types may be wrong on heterogeneous columns.
• Predicate operator set is fixed: ==, !=, <, <=, >, >=.

Avro

rugo.avro reads and writes Avro object container files — the format of Kafka Connect sinks and of Iceberg's manifests. The decoder is native C++: each file's schema is compiled once into a decode program, and values are written straight into Draken columns — no Python object per value.

Quick start

from rugo import avro

meta = avro.read_metadata("launches.avro")       # header only: schema, codec, metadata
print(meta.codec, meta.columns)

with avro.read_avro("launches.avro", columns=["Company", "Rocket.Name"]) as reader:
    for morsel in reader:                         # one Morsel per batch (whole blocks, <= 65,536 rows)
        print(morsel.column(b"Rocket.Name").to_pylist())

read_avro

Types: boolean BOOL · int INT32 · long INT64 · float/double FLOAT32/FLOAT64 · string VARCHAR · bytes/fixed VARBINARY · enum VARCHAR (one entry per symbol, a position per row) · date DATE · time-* TIME (µs) · timestamp-* TIMESTAMP (µs, UTC) · decimal DECIMAL / DECIMAL128 · an array of a plain scalar ARRAY · a whole record, map, or array of nested values NVARCHAR JSON text.

Codecs: null, deflate, snappy, zstandard, bzip2.

write_avro

data = avro.write_avro(morsel, codec="deflate")   # -> bytes; codec null / deflate / snappy / zstandard / bzip2

A column with no nulls is written as plain T, one with nulls as ["null", T]; the root record is rugo.record (record_name= to change it). Types: BOOL boolean · INT8/16/32, UINT8/16 int · INT64, UINT32 long · FLOAT32 float · FLOAT64 double · VARCHAR/NVARCHAR/VARIANT string · VARBINARY bytes · DATE date · TIME time-millis / time-micros · TIMESTAMP timestamp-millis / timestamp-micros (by the column's unit) · DECIMAL fixed + decimal(p, s) at the smallest width for p · UUID fixed(16) + uuid · IPV4 long with an "opteryx.type": "IPV4" attribute, which rugo's reader turns back into IPV4 and every other reader ignores · ARRAY array of ["null", T], nested arrays nested. With dictionary=True (the default) a VARCHAR / NVARCHAR column whose values are all valid Avro names, with at most 4,096 distinct values, is written as an enum when that is smaller; rugo reads it back dictionary-encoded.

open_avro_writer(sink) / write_avro_stream(morsels, sink) stream batches to a callable (e.g. f.write). A stream cannot see later batches, so its fields keep ["null", T] and its strings stay strings; the first batch fixes the schema. Rows are packed into blocks of block_bytes (default 1 MiB) encoded bytes. Refused, with an error naming the column: UINT64, INTERVAL, VECTOR_FP16 and nanosecond TIME / TIMESTAMP (Avro cannot carry them), and a column name that is not a valid Avro name ([A-Za-z_][A-Za-z0-9_]* — nothing is renamed). Design: docs/AVRO_WRITER_DESIGN.md.

Performance

Decoding 1,000,000 rows × 20 columns (dev/bench_avro_readers.py, Apple M-series, medians of interleaved rounds):

The deflate row uses libdeflate (after 0.5.0; 0.72 s with miniz in 0.5.0). rugo returns columns; fastavro and Apache return a dict per record. Converting rugo's columns to Python lists as well still leaves it 4–6× faster than fastavro on these files.

Limitations

Refused, with an error naming what was refused: the xz codec, unions other than ["null", T], recursive types, uuid, decimal precision above 38, timestamp-nanos, local-timestamp-*, duration, and alias-based schema resolution. A file is decoded on one thread. At the Python edge a reader-schema field the file lacks cannot be a constant TIME/TIMESTAMP/DECIMAL column (the Opteryx engine builds those natively).

Design notes

• No PyArrow, no NumPy. Every read and write path is pure C++/Cython and Draken-native. Output Parquet is still standard and PyArrow-readable.
• Fail loud. can_decode(...) is a quick compatibility signal, not a guarantee; on partial decode failure a selected column may be returned as None — check, don't assume success.
• Read less. The advantage over bulk readers comes from projection and predicate/row-group pruning, not raw scan throughput.

Example notebook

space_missions.ipynb walks through a complete workflow on a real dataset:

• Download a Parquet file and inspect its schema with read_metadata
• Filter launches by company with row-group pruning and row-level predicate
• Aggregate total spend per company across streaming morsels
• Write filtered results to JSONL and read them back

Building from source

End users should pip install rugo and use the published wheels. To build from the opteryx-core source tree (Rugo is developed there alongside Draken and the Opteryx engine):

python rugo/setup.py bdist_wheel     # build the standalone Rugo wheel (from repo root)

For in-place development of the whole tree, use the repository's make compile.

License

Apache-2.0. Rugo is part of the Opteryx project.