Skip to main content

A tool to convert CSVs to Parquet files

Project description

csv2parquet

Build Status codecov

Convert a CSV to a parquet file. You may also find sqlite-parquet-vtable useful.

Installing

If you just want to use the tool:

sudo pip install pyarrow csv2parquet

If you want to clone the repo and work on the tool, install its dependencies via pipenv:

pipenv install

Usage

Next, create some Parquet files. The tool supports CSV and TSV files.

usage: csv2parquet [-h] [-n ROWS] [-r ROW_GROUP_SIZE] [-o OUTPUT] [-c CODEC]
                   [-i INCLUDE [INCLUDE ...] | -x EXCLUDE [EXCLUDE ...]]
                   [-R RENAME [RENAME ...]] [-t TYPE [TYPE ...]]
                   csv_file

positional arguments:
  csv_file              input file, can be CSV or TSV

optional arguments:
  -h, --help            show this help message and exit
  -n ROWS, --rows ROWS  The number of rows to include, useful for testing.
  -r ROW_GROUP_SIZE, --row-group-size ROW_GROUP_SIZE
                        The number of rows per row group.
  -o OUTPUT, --output OUTPUT
                        The parquet file
  -c CODEC, --codec CODEC
                        The compression codec to use (brotli, gzip, snappy,
                        zstd, none)
  -i INCLUDE [INCLUDE ...], --include INCLUDE [INCLUDE ...]
                        Include the given columns (by index or name)
  -x EXCLUDE [EXCLUDE ...], --exclude EXCLUDE [EXCLUDE ...]
                        Exclude the given columns (by index or name)
  -R RENAME [RENAME ...], --rename RENAME [RENAME ...]
                        Rename a column. Specify the column to be renamed and
                        its new name, eg: 0=age or person_age=age
  -t TYPE [TYPE ...], --type TYPE [TYPE ...]
                        Parse a column as a given type. Specify the column and
                        its type, eg: 0=bool? or person_age=int8. Parse errors
                        are fatal unless the type is followed by a question
                        mark. Valid types are string (default), bool, int8,
                        int16, int32, int64, float32, float64, timestamp

Testing

pylint csv2parquet
pytest

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for csv2parquet, version 0.0.6
Filename, size & hash File type Python version Upload date
csv2parquet-0.0.6-py3-none-any.whl (7.7 kB) View hashes Wheel py3
csv2parquet-0.0.6.tar.gz (6.2 kB) View hashes Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page