parq-tools 0.1.0

No project description provided

parq-tools

Overview

parq-tools is a collection of utilities for efficiently working with large-scale Parquet datasets. Designed for scalability, it supports chunk-wise processing, metadata handling, and optimized workflows for datasets too large to fit into memory.

Features

• Filtering → Efficiently filter large parquet files.
• Concatenation → Combines multiple Parquet files efficiently along rows (axis=0) or columns (axis=1).
• Tokenized Filtering → Converts pandas-style expressions into efficient PyArrow queries.
• Block Model Generation → Creates massive Parquet datasets that exceed memory limits, useful for testing pipelines.
• Profiling Enhancements → Improves ydata-profiling by profiling specific columns incrementally, merging results for large files.