s3splitmerge 0.0.1

Features

Split:

• split big data file on (>=500MB) in common data format CSV, TSV, JSON into

Install

pip install awswrangler==2.10.0 --no-deps

s3splitmerge is released on PyPI, so all you need is:

$ pip install s3splitmerge

To upgrade to latest version:

$ pip install --upgrade s3splitmerge

Merge Multiple AWS S3 Json File into One Big

1. Input Data

Files:

s3://my-bucket/input/date=2000-01-01/a.json # 6MB
s3://my-bucket/input/date=2000-01-01/b.json # 600MB
s3://my-bucket/input/date=2000-01-01/c.json # 120MB
s3://my-bucket/input/date=2000-01-02/...
...

Content:

{"id": 1, "value": "a", ...}
{"id": 2, "value": "b", ...}
{"id": 3, "value": "c", ...}

2. Normalize file size to approximately 6MB. If smaller than 6MB, keep it as it is:

   s3://my-bucket/input-normalized/date=2000-01-01/a-1.json # 6MB
   
   s3://my-bucket/input-normalized/date=2000-01-01/b-1.json # 6MB
   s3://my-bucket/input-normalized/date=2000-01-01/b-2.json # 6MB
   ...
   s3://my-bucket/input-normalized/date=2000-01-01/b-100.json # 6MB
   
   s3://my-bucket/input-normalized/date=2000-01-01/c-1.json # 6MB
   s3://my-bucket/input-normalized/date=2000-01-01/c-2.json # 6MB
   ...
   s3://my-bucket/input-normalized/date=2000-01-01/c-20.json # 6MB

3. Performance per file ETL using AWS Lambda:

   s3://my-bucket/output-normalized/date=2000-01-01/a-1.parquet # 6MB
   
   s3://my-bucket/output-normalized/date=2000-01-01/b-1.parquet # 6MB
   s3://my-bucket/output-normalized/date=2000-01-01/b-2.parquet # 6MB
   ...
   s3://my-bucket/output-normalized/date=2000-01-01/b-100.parquet # 6MB
   
   s3://my-bucket/output-normalized/date=2000-01-01/c-1.parquet # 6MB
   s3://my-bucket/output-normalized/date=2000-01-01/c-2.parquet # 6MB
   ...
   s3://my-bucket/output-normalized/date=2000-01-01/c-20.parquet # 6MB

4. Merge file into Bigger one for better Athena Query performance:

   s3://my-bucket/output-normalized/date=2000-01-01/part-1.parquet # 500MB
   s3://my-bucket/output-normalized/date=2000-01-01/part-2.parquet # 500MB
   ...