parquet-tools 0.2.3

Easy install parquet-tools

parquet-tools

This is a pip installable parquet-tools. In other words, parquet-tools is a CLI tools of Apache Arrow. You can show parquet file content/schema on local disk or on Amazon S3.

Features

• Read Parquet data (local file or file on S3)
• Read Parquet metadata/schema (local file or file on S3)

Installation

$ pip install parquet-tools

Usage

$ parquet-tools --help
usage: parquet-tools [-h] {show,csv,inspect} ...

parquet CLI tools

positional arguments:
  {show,csv,inspect}
    show              Show human readble format. see `show -h`
    csv               Cat csv style. see `csv -h`
    inspect           Inspect parquet file. see `inspect -h`

optional arguments:
  -h, --help          show this help message and exit

Usage Examples

Show local parquet file

$ parquet-tools show test.parquet
+-------+-------+---------+
|   one | two   | three   |
|-------+-------+---------|
|  -1   | foo   | True    |
| nan   | bar   | False   |
|   2.5 | baz   | True    |
+-------+-------+---------+

Show parquet file on S3

$ parquet-tools show s3://bucket-name/prefix/*
+-------+-------+---------+
|   one | two   | three   |
|-------+-------+---------|
|  -1   | foo   | True    |
| nan   | bar   | False   |
|   2.5 | baz   | True    |
+-------+-------+---------+

Inspect parquet file schema

FileMetaData
■■■■version = 1
■■■■schema = list
...
■■■■column_orders = list
■■■■■■■■ColumnOrder
■■■■■■■■■■■■TYPE_ORDER = TypeDefinedOrder
■■■■■■■■ColumnOrder
■■■■■■■■■■■■TYPE_ORDER = TypeDefinedOrder
■■■■■■■■ColumnOrder
■■■■■■■■■■■■TYPE_ORDER = TypeDefinedOrder
■■■■■■■■ColumnOrder
■■■■■■■■■■■■TYPE_ORDER = TypeDefinedOrder

Cat CSV parquet and transform csvq

$ parquet-tools csv s3://bucket-name/test.parquet |csvq "select one, three where three"
+-------+-------+
|  one  | three |
+-------+-------+
| -1.0  | True  |
| 2.5   | True  |
+-------+-------+