Skip to main content

Pandas ExtensionDType/Array backed by Apache Arrow

Project description

fletcher

CI Code style: black Binder

A library that provides a generic set of Pandas ExtensionDType/Array implementations backed by Apache Arrow. They support a wider range of types than Pandas natively supports and also bring a different set of constraints and behaviours that are beneficial in many situations.

Usage

To use fletcher in Pandas DataFrames, all you need to do is to wrap your data in a FletcherChunkedArray or FletcherContinuousArray object. Your data can be of either pyarrow.Array, pyarrow.ChunkedArray or a type that can be passed to pyarrow.array(…).

import fletcher as fr
import pandas as pd

df = pd.DataFrame({
    'str_chunked': fr.FletcherChunkedArray(['a', 'b', 'c']),
    'str_continuous': fr.FletcherContinuousArray(['a', 'b', 'c']),
})

df.info()

# <class 'pandas.core.frame.DataFrame'>
# RangeIndex: 3 entries, 0 to 2
# Data columns (total 2 columns):
#  #   Column          Non-Null Count  Dtype                      
# ---  ------          --------------  -----                      
#  0   str_chunked     3 non-null      fletcher_chunked[string]   
#  1   str_continuous  3 non-null      fletcher_continuous[string]
# dtypes: fletcher_chunked[string](1), fletcher_continuous[string](1)
# memory usage: 166.0 bytes

Development

While you can use fletcher in pip-based environments, we strongly recommend using a conda based development setup with packages from conda-forge.

# Create the conda environment with all necessary dependencies
conda env create

# Activate the newly created environment
conda activate fletcher

# Install fletcher into the current environment
python -m pip install -e . --no-build-isolation --no-use-pep517

# Run the unit tests (you should do this several times during development)
py.test -nauto

# Install pre-commit hooks
# These will then be automatically run on every commit and ensure that files
# are black formatted, have no flake8 issues and mypy checks the type consistency.
pre-commit install

Code formatting is done using black. This should keep everything in a consistent styling and the formatting is automatically adjusted via the pre-commit hooks.

Using pandas in development mode

To test and develop against pandas' master or your local fixes, you can install a development version of pandas using:

git clone https://github.com/pandas-dev/pandas
cd pandas

# Install additional pandas dependencies
conda install -y cython

# Build and install pandas
python setup.py build_ext --inplace -j 4
python -m pip install -e . --no-build-isolation --no-use-pep517

This links the development version of pandas into your fletcher conda environment. If you change any Python code in pandas, it is directly reflected in your environment. If you change any Cython code in pandas, you need to re-execute python setup.py build_ext --inplace -j 4.

Using (py)arrow nightlies

To test and develop against the latest development version of Apache Arrow (pyarrow), you can install it from the arrow-nightlies conda channel:

conda install -c arrow-nightlies arrow-cpp pyarrow

Benchmarks

In benchmarks/ we provide a set of benchmarks to compare the performance of fletcher against pandas and ensure that fletcher itself stays performant. The benchmarks are written using airspeed velocity. When developing the benchmarks you can run them using asv dev (use -b <pattern> to only run a selection of them) only once. To get real benchmark values, you should use asv run --python=same to run the benchmarks multiple times and get meaningful average runtimes.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for fletcher, version 0.7.2
Filename, size File type Python version Upload date Hashes
Filename, size fletcher-0.7.2-py3-none-any.whl (49.8 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size fletcher-0.7.2.tar.gz (72.7 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page