parquet-analyzer 0.2.1

Inspect the on-disk layout and metadata of Parquet files.

Parquet Analyzer

A Python tool for deep inspection and analysis of Apache Parquet files, providing detailed insights into file structure, metadata, and binary layout.

Installation

pip install parquet-analyzer

To work from a local clone instead, install in editable mode:

pip install -e .

Requirements

• Python 3.11+
• thrift>=0.16 (installed automatically)

Usage

Basic usage

# Analyze a Parquet file and emit the JSON summary/footer/pages bundle
parquet-analyzer example.parquet

# Show raw segment structures (offsets, lengths, thrift payloads)
parquet-analyzer --output-mode segments example.parquet

# Generate an interactive HTML report and save it to disk
parquet-analyzer --output-mode html -o report.html example.parquet

# Enable debug logging while running any mode
parquet-analyzer --log-level DEBUG example.parquet

# Run via python -m if the console script is unavailable
python -m parquet_analyzer example.parquet

Output Formats

Standard output (--output-mode default)

The default output provides a structured JSON payload with three main sections:

Summary statistics

{
  "summary": {
    "num_rows": 10,
    "num_row_groups": 1,
    "num_columns": 2,
    "num_pages": 2,
    "num_data_pages": 2,
    "num_v1_data_pages": 2,
    "num_v2_data_pages": 0,
    "num_dict_pages": 0,
    "page_header_size": 47,
    "uncompressed_page_data_size": 130,
    "compressed_page_data_size": 96,
    "uncompressed_page_size": 177,
    "compressed_page_size": 143,
    "column_index_size": 48,
    "offset_index_size": 23,
    "bloom_filter_size": 0,
    "footer_size": 527,
    "file_size": 753
  }
}

Footer metadata

Complete Parquet file metadata including:

• Schema definition with column types and repetition levels
• Row group information
• Column chunk metadata
• Encoding and compression details

Page information

Detailed breakdown of all pages organized by column:

• Data pages with encoding and statistics
• Dictionary pages
• Column indexes
• Offset indexes
• Bloom filters

Detailed segments (--output-mode segments)

When using --output-mode segments, the tool outputs a detailed segment-by-segment breakdown showing:

[
  {
    "offset": 0,
    "length": 4,
    "name": "magic_number",
    "value": "PAR1"
  },
  {
    "offset": 4,
    "length": 24,
    "name": "page",
    "value": [
      {
        "offset": 5,
        "length": 1,
        "name": "type",
        "value": 0,
        "metadata": {
          "type": "i32",
          "enum_type": "PageType",
          "enum_name": "DATA_PAGE"
        }
      }
    ]
  }
]

This mode is useful for:

• Debugging Parquet file corruption
• Understanding exact binary layout
• Analyzing file format compliance
• Optimizing file structure

HTML report (--output-mode html)

Emits a standalone HTML document with collapsible sections for summary statistics, schema, key-value metadata, row groups, aggregated column statistics, segments, and (optionally) the raw footer. Use the --html-sections flag to control which sections are rendered:

parquet-analyzer --output-mode html \
  --html-sections summary schema columns \
  -o report.html \
  example.parquet

Technical details

The tool uses a custom Thrift protocol implementation (OffsetRecordingProtocol) that wraps the standard Thrift compact protocol to track byte offsets and lengths of all decoded structures. This enables precise mapping of logical Parquet structures to their binary representation.

Development

Environment setup

pip install -e .[dev]
hatch run dev:lint
hatch run dev:test
hatch run dev:test-cov
# Or run everything at once
hatch run dev:check

The development extra pulls in tooling (hatch, ruff, pytest) and pyarrow so tests can generate Parquet fixtures on the fly.

Regenerating Thrift bindings

The Python modules in src/parquet are generated from parquet.thrift.

1. Install the Apache Thrift compiler (brew install thrift on macOS, or download a release from the Apache Thrift project).

2. From the repository root, regenerate everything in one step:

   hatch run dev:update-thrift
   

   This refreshes parquet.thrift, runs the compiler, and removes any stray src/__init__.py the compiler may create.

Contributing

Contributions are welcome! Please feel free to submit issues, feature requests, or pull requests.

License

Released under the MIT License.