parq-tools 0.3.0

A collection of utilities for efficiently working with **large-scale** Parquet datasets.

parq-tools

Overview

parq-tools is a collection of utilities for efficiently working with large-scale Parquet datasets. A typical use case is asset-based workflows with large scientific datasets.

:::note If your datasets are not large, you might find the pandas library more convenient. :::

Features

• Filtering → Efficiently filter large parquet files.
• Concatenation → Combines multiple Parquet files efficiently along rows (axis=0) or columns (axis=1).
• Tokenized Filtering → Converts pandas-style expressions into efficient PyArrow queries.
• Profiling Enhancements → Improves ydata-profiling by profiling specific columns incrementally, merging results for large files.
• Block Model Generation → Create a parquet block model that exceeds the machine memory capacity, useful for testing pipelines.