Chunking for Pandas

A Python package to chunk pandas/numpy data with different chunking strategies. PyPI package found here: https://pypi.org/project/chunking-pandas/

Requirements

• Python 3.10+
• Gradio
• pandas
• pytest
• numpy
• seaborn
• matplotlib
• pyarrow

Installation

pip install chunking-pandas

Or:

make install

Usage

from chunking_pandas import ChunkingExperiment

Create an instance of a Chunking class

class_instance = ChunkingExperiment(
"input.csv",
"output.csv",
n_chunks=3,
chunking_strategy="rows"
)

Then perform the chunking:

class_instance.process_chunks()

Run the web interface

To run as a command, either run:

make run-app

or as a module:

python -m chunking_pandas

Or as a console script:

chunking-interface

Or to run the app programmatically:

from chunking_pandas import launch_interface
launch_interface()

Features

• Multiple chunking strategies (rows, columns, tokens, blocks, parallel strategies)
• Support for CSV, JSON, Numpy and Parquet files
• Web interface using Gradio
• Comprehensive test suite
• Documentation using Sphinx
• Benchmarking various chunking strategies
• Basic examples contained within examples/

Development

To install development dependencies:

pip install -e .[dev]

Or:

make install-dev

Testing

To run tests, run the following command from the root folder:

pytest

Documentation

To install the documentation dependencies:

pip install -e .[docs]

Or:

make install-docs

To build the documentation:

make docs

To serve the documentation:

make docs-serve

Below are the full list of commands that can be run from the root folder:

make benchmark
make clean
make docs
make docs-clean
make docs-serve
make install
make install-docs
make lint
make run-app
make test
make typecheck

License

This project is licensed under the MIT License - see the LICENSE file for details.