Json2Parquet 
This library wraps pyarrow to provide some tools to easily convert JSON data into Parquet format. It is mostly in Python. It iterates over files. It copies the data several times in memory. It is not meant to be the fastest thing available. However, it is convenient for smaller data sets, or people who don’t have a huge issue with speed.
Installation
With pip:
pip install json2parquetWith conda:
conda install -c conda-forge json2parquetUsage
Here’s how to load a random JSON dataset.
from json2parquet import convert_json
# Infer Schema (requires reading dataset for column names)
convert_json(input_filename, output_filename)
# Given columns
convert_json(input_filename, output_filename, ["my_column", "my_int"])
# Given columns and custom field names
field_aliases = {'my_column': 'my_updated_column_name', "my_int": "my_integer"}
convert_json(input_filename, output_filename, ["my_column", "my_int"], field_aliases=field_aliases)
# Given PyArrow schema
import pyarrow as pa
schema = pa.schema([
pa.field('my_column', pa.string),
pa.field('my_int', pa.int64),
])
convert_json(input_filename, output_filename, schema)You can also work with Python data structures directly
from json2parquet import load_json, ingest_data, write_parquet, write_parquet_dataset
# Loading JSON to a PyArrow RecordBatch (schema is optional as above)
load_json(input_filename, schema)
# Working with a list of dictionaries
ingest_data(input_data, schema)
# Working with a list of dictionaries and custom field names
field_aliases = {'my_column': 'my_updated_column_name', "my_int": "my_integer"}
ingest_data(input_data, schema, field_aliases)
# Writing Parquet Files from PyArrow Record Batches
write_parquet(data, destination)
# You can also pass any keyword arguments that PyArrow accepts
write_parquet(data, destination, compression='snappy')
# You can also write partitioned date
write_parquet_dataset(data, destination_dir, partition_cols=["foo", "bar", "baz"])If you know your schema, you can specify custom datetime formats (only one for now). This formatting will be ignored if you don’t pass a PyArrow schema.
from json2parquet import convert_json
# Given PyArrow schema
import pyarrow as pa
schema = pa.schema([
pa.field('my_column', pa.string),
pa.field('my_int', pa.int64),
])
date_format = "%Y-%m-%dT%H:%M:%S.%fZ"
convert_json(input_filename, output_filename, schema, date_format=date_format)Although json2parquet can infer schemas, it has helpers to pull in external ones as well
from json2parquet import load_json
from json2parquet.helpers import get_schema_from_redshift
# Fetch the schema from Redshift (requires psycopg2)
schema = get_schema_from_redshift(redshift_schema, redshift_table, redshift_uri)
# Load JSON with the Redshift schema
load_json(input_filename, schema)Operational Notes
If you are using this library to convert JSON data to be read by Spark, Athena, Spectrum or Presto make sure you use use_deprecated_int96_timestamps when writing your Parquet files, otherwise you will see some really screwy dates.
Contributing
Code Changes
Clone a fork of the library
Run make setup
Run make test
Apply your changes (don’t bump version)
Add tests if needed
Run make test to ensure nothing broke
Submit PR
Documentation Changes
It is always a struggle to keep documentation correct and up to date. Any fixes are welcome. If you don’t want to clone the repo to work locally, please feel free to edit using Github and to submit Pull Requests via Github’s built in features.
1 maintainer
Release files for json2parquet 2.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| json2parquet-2.2.0.tar.gz | 10.5 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| json2parquet-2.2.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 18.3 kB
Release files / json2parquet-2.2.0.tar.gz
| Download URL | json2parquet-2.2.0.tar.gz |
|---|---|
| Size | 10.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b40b2d6e2d98c6fe01a5b35e1a0d6685e24200b237c7e69ea64c00a36f555e59
|
|
BLAKE2b-256 checksum How to use checksums |
ad9af89cf9347e1c3bf3d93fc5a37495ddd5b7d0f04a916d59598b52bfed6044
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.3 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.16
|
Release files / json2parquet-2.2.0-py3-none-any.whl
| Download URL | json2parquet-2.2.0-py3-none-any.whl |
|---|---|
| Size | 7.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c0c2d458e15805e369445bfbec0461fc102380a5953f3c0c0ace87256710d6ce
|
|
BLAKE2b-256 checksum How to use checksums |
1466c27e1c0db2299ab437284933ee63de4fb40c35dbb1b3a15028b8c4758351
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.8 tqdm/4.63.0 importlib-metadata/4.11.3 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.16
|
