Python support for Parquet file format
Project description
fastparquet is a python implementation of the parquet format, aiming integrate into python-based big data work-flows. It is used implicitly by the projects Dask, Pandas and intake-parquet.
We offer a high degree of support for the features of the parquet format, and very competitive performance, in a small install size and codebase.
Details of this project, how to use it and comparisons to other work can be found in the documentation.
Requirements
(all development is against recent versions in the default anaconda channels and/or conda-forge)
Required:
numpy
pandas
cython >= 0.29.23 (if building from pyx files)
cramjam
fsspec
Supported compression algorithms:
Available by default:
gzip
snappy
brotli
lz4
zstandard
Optionally supported
Installation
Install using conda, to get the latest compiled version:
conda install -c conda-forge fastparquet
or install from PyPI:
pip install fastparquet
You may wish to install numpy first, to help pip’s resolver. This may install an appropriate wheel, or compile from source. For the latter, you will need a suitable C compiler toolchain on your system.
You can also install latest version from github:
pip install git+https://github.com/dask/fastparquet
in which case you should also have cython to be able to rebuild the C files.
Usage
Please refer to the documentation.
Reading
from fastparquet import ParquetFile
pf = ParquetFile('myfile.parq')
df = pf.to_pandas()
df2 = pf.to_pandas(['col1', 'col2'], categories=['col1'])
You may specify which columns to load, which of those to keep as categoricals (if the data uses dictionary encoding). The file-path can be a single file, a metadata file pointing to other data files, or a directory (tree) containing data files. The latter is what is typically output by hive/spark.
Writing
from fastparquet import write
write('outfile.parq', df)
write('outfile2.parq', df, row_group_offsets=[0, 10000, 20000],
compression='GZIP', file_scheme='hive')
The default is to produce a single output file with a single row-group (i.e., logical segment) and no compression. At the moment, only simple data-types and plain encoding are supported, so expect performance to be similar to numpy.savez.
History
This project forked in October 2016 from parquet-python, which was not designed for vectorised loading of big data or parallel access.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for fastparquet-2024.5.0-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cd3473d3e299bfb04c0ac7726cca5d13ee450cc2387ee7fd70587ca150647315 |
|
MD5 | d9750fe21ec2334cac35c92cf27e4419 |
|
BLAKE2b-256 | 384ee5969a22b9ef1a7095c9dd37de5614b0c497fe36343ad707b8ae15b52ea5 |
Hashes for fastparquet-2024.5.0-cp312-cp312-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2a951106782d51e5ab110beaad29c4aa0537f045711bb0bf146f65aeaed14174 |
|
MD5 | 4b4ee8e100135a7a091e7c3734d837a6 |
|
BLAKE2b-256 | 23a4b51b04c6a8e7c15c896316f3f0206d5e1e1c0572c5f776cc0ff88f464095 |
Hashes for fastparquet-2024.5.0-cp312-cp312-musllinux_1_2_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c8b2e86fe6488cce0e3d41263bb0296ef9bbb875a2fca09d67d7685640017a66 |
|
MD5 | fa2a0563ddabbc1d3e8964c6a85d84b4 |
|
BLAKE2b-256 | 7dce27a706855951cc4d1af74515650649bbc43b3947ed7c7fed5fefb8bbd066 |
Hashes for fastparquet-2024.5.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2b3cf7b4eb1b06e87b97a3a5c9124e4b1c08a8903ba017052c5fe2c482414a3d |
|
MD5 | 34155f30f478ffa18401068c7ce23953 |
|
BLAKE2b-256 | 5d5a5dc0de7208f33bcd3223836614eab9ba1cfcd69147010827dab1eabea573 |
Hashes for fastparquet-2024.5.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4e6ac308a2f391ce589c99b8376e7cdfe4241ef5770ac4cf4c1c93f940bda83c |
|
MD5 | e7c0b729e74c4408b2d11e7495fc667a |
|
BLAKE2b-256 | ba82746e1aefd3dce0c3ef9214d3fae7d675728661294e74016d2bb4c4b4371c |
Hashes for fastparquet-2024.5.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5626fc72204001b7e82fedb4b02174ecb4e2d4143b38b4ea8d2f9eb65f6b000e |
|
MD5 | 213ffc94ef57dbcf3635961fdc2a3d59 |
|
BLAKE2b-256 | 4411d7d7c539992b715449cd8615f2bb06cdd06157218b7f705d51ade7d3a0f4 |
Hashes for fastparquet-2024.5.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3a60f7b0b308d6b9f12c642cf5237a05d754926fb31ce865ff7072bceab19fbb |
|
MD5 | 45ee4a31a04c20b8f7c0536ed2fd0c5a |
|
BLAKE2b-256 | 63797ccc7ad2c215b5468d3fd4a611ca851d94332e16a288e28b073a21c81bf2 |
Hashes for fastparquet-2024.5.0-cp312-cp312-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e1fa068ef1826bff6d4a9106a6f9e9d6fd20b8b516da4b82d87840cb5fd3947c |
|
MD5 | 0db366f0b18e14c0c63237a82ed8beff |
|
BLAKE2b-256 | 3c4e1068ebf0482bac3bfc96209457f952ae5115422763818decec15c948edec |
Hashes for fastparquet-2024.5.0-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ea3796c4a38ef8b372a3056b5cef52ca8182fa554fa51c7637c2421e69ee56e5 |
|
MD5 | 1cf344cb357f0dd8ab3ad3c8cdabf608 |
|
BLAKE2b-256 | 6f728047d08e8717e8b201dc87059776ddcaf15863d7a8c4eb5adf43e87ee00a |
Hashes for fastparquet-2024.5.0-cp311-cp311-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1496d83d7a77c19abae796e3b582539884fc893d75a3ad4f90df12f8f23a902a |
|
MD5 | 4385d1b43e40f3b24e2775f75dfa7d32 |
|
BLAKE2b-256 | 9192a62413458ed37f5f45b6f93dfd6b6a78f1fd7da8220f1641422ed277f216 |
Hashes for fastparquet-2024.5.0-cp311-cp311-musllinux_1_2_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ba251231b005c0f3f7e56f6e9cd1939be99b2d810ab5b05039271e260c0196c6 |
|
MD5 | cef34e6f0577fde62fbbbc81e35c8b39 |
|
BLAKE2b-256 | 404c9e1406ba1ea3397b288027160735691c66adbc7ac8e7e29b2fabd1a8bb2d |
Hashes for fastparquet-2024.5.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 08358d99278c5d3fb523d819fff5c74d572d8f67ebbe2215a2c7bfca7e3664cf |
|
MD5 | 06fb2c000ef604974bb04335f09e8a86 |
|
BLAKE2b-256 | 6e72ff30ab77b4aa4d76f07837a72169764260b2ee221d2e2fa598a9a945334a |
Hashes for fastparquet-2024.5.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d20bba5c39139a88d8d6931764b830ba14042742d802238d9edf86d4d765ad7a |
|
MD5 | 6ab38b3961f67dc5e07799dc34236f82 |
|
BLAKE2b-256 | ec3e71045851a2d8d3bdbaa197ce6ae648fb351ac88b28d5fa9983e26db3a162 |
Hashes for fastparquet-2024.5.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e9de270e17a6ae2f02c716421d60e18d35d4718037f561b3e359989db19f700a |
|
MD5 | dd8151dd2575ff9e5b69f1962b928e52 |
|
BLAKE2b-256 | d2a87bb621ac42fc476ed91b6e442674fef7e79e1f99567b34a3c937d2b63e6f |
Hashes for fastparquet-2024.5.0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 784989ee2c251960b8f00dc38c6c730f784712c8e3d08cc7e0ce842055476af1 |
|
MD5 | c43e1dd1bf3398878f72036497fe5e38 |
|
BLAKE2b-256 | 6f898564d2e0e670aef997d02e9f67a9c7ab86772e423295a33f99578b0dab70 |
Hashes for fastparquet-2024.5.0-cp311-cp311-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 56d03b0a291d6a575ab365516c53b4da8e040347f8d43af79be25893c591b38c |
|
MD5 | 05e02d5a667d3a16c5b92f88e80f40a6 |
|
BLAKE2b-256 | f0d18f95110eb099dd630a756b1cd8b480c5a42715cd2ca481bfac48f38b0c4d |
Hashes for fastparquet-2024.5.0-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f5c3cabcfa2f534e4b23343c1ab84c37d336da73770005e608d1894ab1084600 |
|
MD5 | eb52e453d319c051689c715371685a08 |
|
BLAKE2b-256 | 42f16cc8fd34494cfd3a9367f006a61f0ce8a1e031678f0e878cdea5410f74dc |
Hashes for fastparquet-2024.5.0-cp310-cp310-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e5b1ed889f4ac7ea059ff95f4a01f5c07c825c50c2e1bc9e2b64c814df94c243 |
|
MD5 | 8fd7bd73a8cf942710bfd03c68ac5c8e |
|
BLAKE2b-256 | 35ff1dfee5868728a93a8033ae6a6ba0c76d7641e1756923fbaf05477570bd7a |
Hashes for fastparquet-2024.5.0-cp310-cp310-musllinux_1_2_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 42149929b71d9122bd501aa695681f40a04a9fa3f5b802cf0fb6aa4e95ccf2dd |
|
MD5 | 196f05c3f02dddbf113e3b07f8d8054d |
|
BLAKE2b-256 | 72843c69c74f629e70e58ae67f8fdc05a376936eb64a3873d0fd471fa541fea5 |
Hashes for fastparquet-2024.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f411056152b5d3cc82b6624d9da80535d10d9277d921fdb2e9516e93c8c227e8 |
|
MD5 | b8ce7acbcd5f0993c744afbdc7b68169 |
|
BLAKE2b-256 | 3ea42ff7b6688d614bef88d2b2a14a5479cbc310d7d296e738d71f182eba2f30 |
Hashes for fastparquet-2024.5.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4a2045c21f90358541286f26f0735bfb2265b075413fbced3b876fc8848eda52 |
|
MD5 | 4a887da0b3fdfe87df2f23585f7050ed |
|
BLAKE2b-256 | cea12f62bf5e1545199451630546484e7d7129534a27b7ad09779ea93fced3ff |
Hashes for fastparquet-2024.5.0-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cc99d7c0f1816394d53aadd47919bba70bb81355259d8788d28e35913816aee0 |
|
MD5 | 8dd5e018f40b5ccffd6b25a9c408d229 |
|
BLAKE2b-256 | 83a1b0d080dac711426ee068152a7cd77323e9ab852b49e0ce1915ca913e1ed5 |
Hashes for fastparquet-2024.5.0-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 07fc5a45450a39cd07c6ef0e0219ac4b1879f8b27c825ee4ba5d87a3ae505f11 |
|
MD5 | d15b80f12ff590bd739042f2f47e66af |
|
BLAKE2b-256 | 795bcece63c1730a05a50b18a5ca20b17370bb2527cc3102516c954ec040266d |
Hashes for fastparquet-2024.5.0-cp310-cp310-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9dfbed87b4b58b0794b2cb3aa4abcb43fc01480a10c7779a323d2dd1599f6acd |
|
MD5 | 2fdea53bebfcb1e57308c4af3fd61ec8 |
|
BLAKE2b-256 | 05d870916170d618ad8dc1445da6ea7cfa0cac7dd22cfa526170c8ed56856d70 |
Hashes for fastparquet-2024.5.0-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cb93e8951f46943c8567c9a555cb3d24d2c78efdf78e95fd72177d80da73a10f |
|
MD5 | d3f373090c09371533701ebc23ef3b5e |
|
BLAKE2b-256 | 5d89858e9ec5efb003b0c412746e275e72563a30772d9d9855e290ee3d147f4e |
Hashes for fastparquet-2024.5.0-cp39-cp39-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 611da9043f9dab1c63e6c90a6b124e3d2789c34fefa00d45356517f1e8a09c83 |
|
MD5 | cfa5786a1d918cb08427c3dff883ddb5 |
|
BLAKE2b-256 | dad2e67d4f080ec12e96a623b2410dd67cefa6b9f16ff07ee886d0b3075473d7 |
Hashes for fastparquet-2024.5.0-cp39-cp39-musllinux_1_2_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b562be0f43a007493014512602ab6b0207d13ea4ae85e0d94d61febf08efa1ee |
|
MD5 | d45c4d6f4997b82c0010c3d6b61ae27a |
|
BLAKE2b-256 | 26c384e721429f032a449f3c1f4bcf819d4b9170e5fd9d39d5d96f03c7a8952c |
Hashes for fastparquet-2024.5.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dfdc8aaec67edd30814c2c2f0e291eb3c3044525d18c87e835ef8793d6e2ea2d |
|
MD5 | af2252ea23eebd93b5e98ab31c7c0976 |
|
BLAKE2b-256 | 3a76622f96e5f7c0409a71853c334fad2ffafe4f09a2699448ebf97834c48c1d |
Hashes for fastparquet-2024.5.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 691348cc85890663dd3c0bb02544d38d4c07a0c3d68837324dc01007301150b5 |
|
MD5 | 6d9e44983670463b788d15992cc36ec8 |
|
BLAKE2b-256 | 4dc6be0fbd7ce7561eb5ff245498180a7f31d92333a599a192f1bd63cb420789 |
Hashes for fastparquet-2024.5.0-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0034d1b5af3a71cc2fb29c590f442c0b514f710d6d6996794ae375dcfe050c05 |
|
MD5 | 1591e5a0ae55f51c2949f6271f202da6 |
|
BLAKE2b-256 | 17b8a49d53dfa80e7a939e4c16a214e443456bd15d6309abce28fcf059debfbc |
Hashes for fastparquet-2024.5.0-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fc3d35ff8341cd65baecac71062e9d73393d7afda207b3421709c1d3f4baa194 |
|
MD5 | ef1d5322a09160c81ecfc7223ede5407 |
|
BLAKE2b-256 | 7a1513b067f7a7785c7ee5232a5edf04ef87637372a35561ff8eba132524866f |
Hashes for fastparquet-2024.5.0-cp39-cp39-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 47695037fdc534ef4247f25ccf17dcbd8825be6ecb70c54ca54d588a794f4a6d |
|
MD5 | d7b630611691ce698aa6cb2f1d95bbdb |
|
BLAKE2b-256 | 6454879c3ae5c3744129e3208b4ceed606a38c92746412e2e4f6b26575839cb2 |