Skip to main content

Python support for Parquet file format

Project description


parquet-python is a pure-python implementation (currently with only read-support) of the parquet format. It comes with a script for reading parquet files and outputting the data to stdout as JSON or TSV (without the overhead of JVM startup). Performance has not yet been optimized, but it’s useful for debugging and quick viewing of data in files.

Not all parts of the parquet-format have been implemented yet or tested e.g. nested data—see Todos below for a full list. With that said, parquet-python is capable of reading all the data files from the parquet-compatability project.


parquet-python has been tested on python 2.7, 3.6, and 3.7. It depends on pythrift2 and optionally on python-snappy (for snappy compressed files, please also install parquet-python[snappy]).

getting started

parquet-python is available via PyPi and can be installed using pip install parquet. The package includes the parquet command for reading python files, e.g. parquet test.parquet. See parquet –help for full usage.


parquet-python currently has two programatic interfaces with similar functionality to Python’s csv reader. First, it supports a DictReader which returns a dictionary per row. Second, it has a reader which returns a list of values for each row. Both function require a file-like object and support an optional columns field to only read the specified columns.

import parquet
import json

## assuming parquet file with two rows and three columns:
## foo bar baz
## 1   2   3
## 4   5   6

with open("test.parquet") as fo:
   # prints:
   # {"foo": 1, "bar": 2}
   # {"foo": 4, "bar": 5}
   for row in parquet.DictReader(fo, columns=['foo', 'bar']):

with open("test.parquet") as fo:
   # prints:
   # 1,2
   # 4,5
   for row in parquet.reader(fo, columns=['foo', 'bar]):
       print(",".join([str(r) for r in row]))


  • Support the deprecated bitpacking

  • Fix handling of repetition-levels and definition-levels

  • Tests for nested schemas, null data

  • Support reading of data from HDFS via snakebite and/or webhdfs.

  • Implement writing

  • performance evaluation and optimization (i.e. how does it compare to the c++, java implementations)


Is done via Pull Requests. Please include tests with your changes and follow pep8.

To run the tests you must install and execute tox (pip install tox) to run for all supported versions. If you want to run just for your current version, execute: pip install -r requirements-development.txt and then nosetests.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

parquet-1.3.1.tar.gz (23.1 kB view hashes)

Uploaded Source

Built Distribution

parquet-1.3.1-py3-none-any.whl (24.6 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page