parquet-tools
This is a pip installable parquet-tools. In other words, parquet-tools is a CLI tools of Apache Arrow. You can show parquet file content/schema on local disk or on Amazon S3. It is incompatible with original parquet-tools.
Features
- Read Parquet data (local file or file on S3)
- Read Parquet metadata/schema (local file or file on S3)
Installation
$ pip install parquet-tools
Usage
$ parquet-tools --help
usage: parquet-tools [-h] {show,csv,inspect} ...
parquet CLI tools
positional arguments:
{show,csv,inspect}
show Show human readble format. see `show -h`
csv Cat csv style. see `csv -h`
inspect Inspect parquet file. see `inspect -h`
optional arguments:
-h, --help show this help message and exit
Usage Examples
Show local parquet file
$ parquet-tools show test.parquet
+-------+-------+---------+
| one | two | three |
|-------+-------+---------|
| -1 | foo | True |
| nan | bar | False |
| 2.5 | baz | True |
+-------+-------+---------+
Show parquet file on S3
$ parquet-tools show s3://bucket-name/prefix/*
+-------+-------+---------+
| one | two | three |
|-------+-------+---------|
| -1 | foo | True |
| nan | bar | False |
| 2.5 | baz | True |
+-------+-------+---------+
Inspect parquet file schema
$ parquet-tools inspect /path/to/parquet
Inspect output
############ file meta data ############
created_by: parquet-cpp version 1.5.1-SNAPSHOT
num_columns: 3
num_rows: 3
num_row_groups: 1
format_version: 1.0
serialized_size: 2226
############ Columns ############
one
two
three
############ Column(one) ############
name: one
path: one
max_definition_level: 1
max_repetition_level: 0
physical_type: DOUBLE
logical_type: None
converted_type (legacy): NONE
############ Column(two) ############
name: two
path: two
max_definition_level: 1
max_repetition_level: 0
physical_type: BYTE_ARRAY
logical_type: String
converted_type (legacy): UTF8
############ Column(three) ############
name: three
path: three
max_definition_level: 1
max_repetition_level: 0
physical_type: BOOLEAN
logical_type: None
converted_type (legacy): NONE
Cat CSV parquet and transform csvq
$ parquet-tools csv s3://bucket-name/test.parquet |csvq "select one, three where three"
+-------+-------+
| one | three |
+-------+-------+
| -1.0 | True |
| 2.5 | True |
+-------+-------+
1 maintainer
Metadata
1 maintainer
Release files for parquet-tools 0.2.16
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| parquet_tools-0.2.16.tar.gz | 28.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| parquet_tools-0.2.16-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 60.1 kB
Release files / parquet_tools-0.2.16.tar.gz
| Download URL | parquet_tools-0.2.16.tar.gz |
|---|---|
| Size | 28.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7bad2ad38f2381afde9d109577199d953946a904b0d74b2f074e47040673f60a
|
|
BLAKE2b-256 checksum How to use checksums |
7155ce056a1be7cb3cf90165240e81cce45848c37a2aa7453156e3229cb79d12
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.18
|
Release files / parquet_tools-0.2.16-py3-none-any.whl
| Download URL | parquet_tools-0.2.16-py3-none-any.whl |
|---|---|
| Size | 31.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
a360eec233123c2c66f38867ec6dd5c79d48b8f757af8162bf8825d0190dd3d5
|
|
BLAKE2b-256 checksum How to use checksums |
60acbcfe4e1a19b382be1cc3106ef64611ef85e8683bb07fcc7eab376170dffe
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.18
|
