rugo 0.1.8

Parquet Metadata Reader

rugo

rugo is a C++17 and Cython powered Parquet metadata reader for Python. It delivers high-throughput metadata inspection without loading columnar data pages.

Key Features

• Fast metadata extraction backed by an optimized C++17 parser and thin Python bindings.
• Complete schema and row-group details, including encodings, codecs, offsets, bloom filter pointers, and custom key/value metadata.
• Works with file paths, byte strings, and contiguous memoryviews for zero-copy parsing.
• Optional schema conversion helpers for Orso.
• No runtime dependencies beyond the Python standard library.

Installation

PyPI

pip install rugo

# Optional extras
pip install rugo[orso]
pip install rugo[dev]

From source

git clone https://github.com/mabel-dev/rugo.git
cd rugo
python -m venv .venv
source .venv/bin/activate
make update
make compile
pip install -e .

Requirements

• Python 3.9 or newer
• A C++17 compatible compiler (clang, gcc, or MSVC)
• Cython and setuptools for source builds (installed by the commands above)

Quickstart

import rugo.parquet as parquet_meta

metadata = parquet_meta.read_metadata("example.parquet")

print(f"Rows: {metadata['num_rows']}")
print("Schema columns:")
for column in metadata["schema_columns"]:
    print(f"  {column['name']}: {column['physical_type']} ({column['logical_type']})")

first_row_group = metadata["row_groups"][0]
for column in first_row_group["columns"]:
    print(
        f"{column['name']}: codec={column['compression_codec']}, "
        f"nulls={column['null_count']}, range=({column['min']}, {column['max']})"
    )

read_metadata returns dictionaries composed of Python primitives, ready for JSON serialisation or downstream processing.

Returned metadata layout

{
    "num_rows": int,
    "schema_columns": [
        {
            "name": str,
            "physical_type": str,
            "logical_type": str,
            "nullable": bool,
        },
        ...
    ],
    "row_groups": [
        {
            "num_rows": int,
            "total_byte_size": int,
            "columns": [
                {
                    "name": str,
                    "path_in_schema": str,
                    "type": str,
                    "logical_type": str,
                    "num_values": Optional[int],
                    "total_uncompressed_size": Optional[int],
                    "total_compressed_size": Optional[int],
                    "data_page_offset": Optional[int],
                    "index_page_offset": Optional[int],
                    "dictionary_page_offset": Optional[int],
                    "min": Any,
                    "max": Any,
                    "null_count": Optional[int],
                    "distinct_count": Optional[int],
                    "bloom_offset": Optional[int],
                    "bloom_length": Optional[int],
                    "encodings": List[str],
                    "compression_codec": Optional[str],
                    "key_value_metadata": Optional[Dict[str, str]],
                },
                ...
            ],
        },
        ...
    ],
}

Fields that are not present in the source Parquet file are reported as None. Minimum and maximum values are decoded into Python types when possible; otherwise hexadecimal strings are returned.

Parsing options

All entry points share the same keyword arguments:

• schema_only (default False): return only the top-level schema without row group details.
• include_statistics (default True): skip min/max/num_values decoding when set to False.
• max_row_groups (default -1): limit the number of row groups inspected; handy for very large files.

metadata = parquet_meta.read_metadata(
    "large_file.parquet",
    schema_only=False,
    include_statistics=False,
    max_row_groups=2,
)

Working with in-memory data

with open("example.parquet", "rb") as fh:
    data = fh.read()

from_bytes = parquet_meta.read_metadata_from_bytes(data)
from_view = parquet_meta.read_metadata_from_memoryview(memoryview(data))

read_metadata_from_memoryview performs zero-copy parsing when given a contiguous buffer.

Prototype Data Decoding (Experimental)

rugo includes a prototype decoder for reading actual column data from Parquet files. This is a limited, experimental feature designed for simple use cases and testing.

Supported Features

• ✅ Uncompressed columns only (codec=UNCOMPRESSED)
• ✅ PLAIN encoding only
• ✅ int32, int64, and string (byte_array) types only

Unsupported Features

• ❌ Compressed columns (SNAPPY, GZIP, ZSTD, etc.)
• ❌ Dictionary encoding, Delta encoding, RLE_DICTIONARY
• ❌ Other types (float, boolean, date, timestamp, complex types)
• ❌ Nullable columns (columns with definition levels)
• ❌ Multiple row groups (only first row group is decoded)

Usage

import rugo.parquet as parquet_meta

# Check if a file can be decoded with the prototype decoder
if parquet_meta.can_decode("data.parquet"):
    # Decode a specific column (returns a Python list)
    values = parquet_meta.decode_column("data.parquet", "column_name")
    print(values)  # e.g., [1, 2, 3, 4, 5] or ['a', 'b', 'c']
else:
    print("File cannot be decoded - use PyArrow or another full decoder")

See examples/decode_example.py for a complete demonstration.

Note: This decoder is a prototype for educational and testing purposes. For production use with full Parquet support, use PyArrow or FastParquet.

Optional Orso conversion

Install the optional extra (pip install rugo[orso]) to enable Orso helpers:

from rugo.converters.orso import extract_schema_only, rugo_to_orso_schema

metadata = parquet_meta.read_metadata("example.parquet")
relation = rugo_to_orso_schema(metadata, "example_table")
schema_info = extract_schema_only(metadata)

See examples/orso_conversion.py for a complete walkthrough.

Development

make update     # install build and test tooling (uses uv under the hood)
make compile    # rebuild the Cython extension with -O3 and C++17 flags
make test       # run pytest-based validation (includes PyArrow comparisons)
make lint       # run ruff, isort, pycln, cython-lint
make mypy       # type checking

make compile clears previous build artefacts before rebuilding the extension in-place.

Project layout

rugo/
├── rugo/__init__.py
├── rugo/parquet/
│   ├── metadata_reader.pyx
│   ├── metadata.cpp
│   ├── metadata.hpp
│   ├── decode.cpp
│   ├── decode.hpp
│   └── thrift.hpp
├── rugo/converters/orso.py
├── examples/
│   ├── comprehensive_metadata.py
│   ├── decode_example.py
│   └── orso_conversion.py
├── tests/
│   ├── data/
│   ├── test_all_metadata_fields.py
│   ├── test_decode.py
│   ├── test_logical_types.py
│   ├── test_orso_converter.py
│   └── test_statistics.py
├── Makefile
├── pyproject.toml
└── README.md

Status and limitations

• Active development status (alpha); API details may evolve.
• Primary focus is metadata inspection; the data decoder is a prototype with limited capabilities.
• Requires a C++17 compiler when installing from source or editing the Cython bindings.
• Bloom filter information is exposed via offsets and lengths; higher-level helpers are planned.

License

Licensed under the Apache License 2.0. See LICENSE for full terms.

Maintainer

Created and maintained by Justin Joyce (@joocer). Contributions are welcome via issues and pull requests.