Introduction
An opinionated JSON to CSV/XLSX/SQLITE/PARQUET converter which tries to make a useful relational output for data analysis.
Web playgroud of CSV/XLSX conversions
Rationale
When receiving a JSON file where the structure is deeply nested or not well specified, it is hard to determine what the data contains. Also, even after knowing the JSON structure, it requires a lot of time to work out how to flatten the JSON into a relational structure to do data analysis on and to be part of a data pipeline.
Flatterer aims to be the first tool to go to when faced with the above problem. It may not be the tool that you end up using to flatten the JSON in your data pipeline, as hand written flattening may be required, but it could be. It has many benefits over most hand written approaches:
- It is fast, written in rust but with python bindings for ease of use. It can be 10x faster than hand written python flattening.
- Memory efficient. Uses a custom streaming JSON parser to mean that long list of objects nested with the JSON will be streamed, so not much data needs to be loaded into memory at once.
- Fast memory efficient output to CSV/XLSX/SQLITE/PARQUET
- Uses best practice that has been learnt from flattening JSON countless times, such as generating keys to link one-to-many tables to their parents.
Install
pip install flatterer
Flatterer requires Python 3.6 or greater. It is written as a python extension in Rust but has binaries (wheels) for linux (x64), macos (x64 and universal) and windows (x64, x86). On other platforms a rust toolchain will need to be installed.
Example JSON
Say you have a JSON data like this named games.json:
[
{
"id": 1,
"title": "A Game",
"releaseDate": "2015-01-01",
"platforms": [
{"name":"Xbox"},
{"name":"Playstation"}
],
"rating": {
"code": "E",
"name": "Everyone"
}
},
{
"id": 2,
"title": "B Game",
"releaseDate": "2016-01-01",
"platforms": [
{"name":"PC"}
],
"rating": {
"code": "E",
"name": "Everyone"
}
}
]
Running Flatterer
Run the above file with flatterer.
flatterer games.json games_dir
Output Files
By running the above you will get the following files:
tree games_dir
games_dir/
├── csv
│ ├── games.csv
│ └── platforms.csv
├── datapackage.json
├── fields.csv
└── ...
Main Table
games.csv contains:
| _link | _link_games | id | rating_code | rating_name | releaseDate | title |
|---|---|---|---|---|---|---|
| 1 | 1 | 1 | E | Everyone | 2015-01-01 | A Game |
| 2 | 2 | 2 | E | Everyone | 2016-01-01 | B Game |
Special column _link is generated. _link is the primary key there unique per game.
Also the rating sub-object is promoted to this table it has a one-to-one relationship with games.
Sub-object properties are separated by '_'.
One To Many Table
platforms is an array so is a one-to-many with games therefore needs its own table:
platforms.csv contains:
| _link | _link_games | name |
|---|---|---|
| 1.platforms.0 | 1 | Xbox |
| 1.platforms.1 | 1 | Playstation |
| 2.platforms.0 | 2 | PC |
Link Fields
_link is the primary key for the platforms table too. Every table except games table, contains a _link_games field to easily join to the main games table.
If there was a sub-array of platforms then that would have _link, _link_games and _link_platforms fields.
To generalize this the _link__<table_name> fields joins to the _link field of <table_name> i.e the _link__<table_name> are the foreign keys refrencing <table_name>._link.
Fields CSV
fields.csv contains some metadata about the output tables:
| table_name | field_name | field_type | count | field_title |
|---|---|---|---|---|
| platforms | _link | text | 3 | _link |
| platforms | _link_games | text | 3 | _link_games |
| platforms | name | text | 3 | name |
| games | _link | text | 2 | _link |
| games | id | number | 2 | id |
| games | rating_code | text | 2 | rating_code |
| games | rating_name | text | 2 | rating_name |
| games | releaseDate | date | 2 | releaseDate |
| games | title | text | 2 | title |
The field_type column contains a type guess useful for inserting into a database. The field_title is the column heading in the CSV file or XLSX tab, which is initally the same as the field_name.
After editing this file then you can rerun the transform:
flatterer games.json new_games_dir -f myfields.csv --only-fields
This can be useful for renameing columns, rearranging the field order or if you want to remove some fields the --only-fields flag will only include the fields in the edited file.
datapackage.json contains metadata in the Tabular Data Package Spec
1 maintainer
Metadata
1 maintainer
Release files for flatterer 0.25.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| flatterer-0.25.0.tar.gz | 8.1 MB | Without provenance | Details |
Built distributions (wheels)
Total release size: 291.8 MB
Release files / flatterer-0.25.0.tar.gz
| Download URL | flatterer-0.25.0.tar.gz |
|---|---|
| Size | 8.1 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e0691fee36c4740fd0328b610d6ce761c6ae4f45314433582148f27bc22a8889
|
|
BLAKE2b-256 checksum How to use checksums |
4124fc03395923c00ff5d1ee21b69b31ec03ccfe10b5fbb6e947100d350e8ee1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl
| Download URL | flatterer-0.25.0-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 14.9 MB |
| Tags | Linux glibc 2.28+ x86-64 PyPy 3.11 PyPy 3.11 7.3 |
|
SHA-256 checksum How to use checksums |
d6c4c8d3752bcc6f811e350de5a0f2d989035a93d9bc9f7ef07113134663c9f7
|
|
BLAKE2b-256 checksum How to use checksums |
86bc784365d205962b2a74597235bca93c8b759d5e8e88215ee03b7ce3306755
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp315-cp315t-manylinux_2_28_x86_64.whl
| Download URL | flatterer-0.25.0-cp315-cp315t-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 14.9 MB |
| Tags | CPython 3.15 CPython 3.15 free-threading Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
a9af1209bce8125363fcb9db9b148a3a653f17c33d829cedb568845e44db9215
|
|
BLAKE2b-256 checksum How to use checksums |
2408c47fac420cd5047525955e9b40836681b34a00f58a0c16eb1508d12ad303
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp315-cp315-manylinux_2_28_x86_64.whl
| Download URL | flatterer-0.25.0-cp315-cp315-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 14.9 MB |
| Tags | CPython 3.15 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
2de737778f93926d115063108db84221d0799240d1ba74d2258e4c0518ede4dc
|
|
BLAKE2b-256 checksum How to use checksums |
f4260a1435ff49ae5689134e1e6668e5512071bc9eefa727efab21f40c247ae6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp314-cp314t-manylinux_2_28_x86_64.whl
| Download URL | flatterer-0.25.0-cp314-cp314t-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 14.9 MB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
ad951b59fac65417b030833a247ddc35dac4c27909a903fb8b378aecc3d9b004
|
|
BLAKE2b-256 checksum How to use checksums |
ce26a3f627746f106c61cdbd7d76c580586ac36944f6c00b4c5196610326b3fd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp314-cp314-win_amd64.whl
| Download URL | flatterer-0.25.0-cp314-cp314-win_amd64.whl |
|---|---|
| Size | 14.0 MB |
| Tags | CPython 3.14 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
a75808ff13a9791f181b9a06e4d1aefd53837366a74fe3df36dd59cf054ed271
|
|
BLAKE2b-256 checksum How to use checksums |
78b7d5afa8b90e49e09f1075b09759897a4fc306383e9d60494ba6e8cd6126d1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp314-cp314-win32.whl
| Download URL | flatterer-0.25.0-cp314-cp314-win32.whl |
|---|---|
| Size | 12.6 MB |
| Tags | CPython 3.14 Windows x86-32 |
|
SHA-256 checksum How to use checksums |
cf1d0fe9d38cf3aefc6b029dfb128f8c2ffe1f553c1443035a47ab0fec0815f9
|
|
BLAKE2b-256 checksum How to use checksums |
767d2904e457db1180363c4bc125cffa8f6364dbd2e76956447ed9d60f9ca7e7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp314-cp314-manylinux_2_28_x86_64.whl
| Download URL | flatterer-0.25.0-cp314-cp314-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 14.9 MB |
| Tags | CPython 3.14 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
c276ced987c3b3ef167545b32ee38bf756dc62f8911f07691c732ca05b129a52
|
|
BLAKE2b-256 checksum How to use checksums |
b57553819108c7c99f930b3cc30dc41eb5430c122491f07e13a61bf615bf0e3b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp314-cp314-macosx_11_0_arm64.whl
| Download URL | flatterer-0.25.0-cp314-cp314-macosx_11_0_arm64.whl |
|---|---|
| Size | 13.2 MB |
| Tags | CPython 3.14 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
e7c8c239ec1e7358d15e6b5e2ded541cce66440dcc16f394690605d5f6ddede9
|
|
BLAKE2b-256 checksum How to use checksums |
47c0c5f1ff49464cda6fdc69e22d9dbfeb141249dc1b565ac3ab841b16b6a680
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp313-cp313-win_amd64.whl
| Download URL | flatterer-0.25.0-cp313-cp313-win_amd64.whl |
|---|---|
| Size | 14.0 MB |
| Tags | CPython 3.13 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
45319a22f5706d7c0896e03fefb2d2284b320e8c3c64b39f4ca956b59364a3ee
|
|
BLAKE2b-256 checksum How to use checksums |
badab45b82473ed6d1c2c23260e25d133a9cc885c89b1e8f0107098981e64fd4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp313-cp313-manylinux_2_28_x86_64.whl
| Download URL | flatterer-0.25.0-cp313-cp313-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 14.9 MB |
| Tags | CPython 3.13 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
0c571ed778e5896bc88d8b450f0b331579b5feb0e6c66f329178a1cd58370177
|
|
BLAKE2b-256 checksum How to use checksums |
1c1134b3cb0f4fedaf2da58f2559e4f783b8235bfb3de13e27961e337b6a9e21
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp313-cp313-macosx_11_0_arm64.whl
| Download URL | flatterer-0.25.0-cp313-cp313-macosx_11_0_arm64.whl |
|---|---|
| Size | 13.2 MB |
| Tags | CPython 3.13 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
57c6acfb481cca2a08b5ab7266cebee9b33164ed75bb078e813e75215e177d6f
|
|
BLAKE2b-256 checksum How to use checksums |
c15fe2a4693cdbe146d55f614ee83a53c7e678ee7304ebca844386776e1bfa62
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp312-cp312-win_amd64.whl
| Download URL | flatterer-0.25.0-cp312-cp312-win_amd64.whl |
|---|---|
| Size | 14.0 MB |
| Tags | CPython 3.12 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
7f743511c8e9483e03df93ab4ea00dd67e592f81c94c2d2e3f4d84ccc44a1bdb
|
|
BLAKE2b-256 checksum How to use checksums |
f661e351a43bbb0b51f47cc8ec15e49d9233698e49a11a291de9152d34f359b2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp312-cp312-manylinux_2_28_x86_64.whl
| Download URL | flatterer-0.25.0-cp312-cp312-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 14.9 MB |
| Tags | CPython 3.12 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
cf745b8a9d798950d65cb18dd7135c3d0d3595e30c67cfb53c85dfd4d1c84f66
|
|
BLAKE2b-256 checksum How to use checksums |
aaa3e3c2c7833e3367e7923abe6c67c117d7df2ba72e49d2a77861b3f322b864
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp312-cp312-macosx_11_0_arm64.whl
| Download URL | flatterer-0.25.0-cp312-cp312-macosx_11_0_arm64.whl |
|---|---|
| Size | 13.2 MB |
| Tags | CPython 3.12 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
e4d38e25d1efd43704702585976d3e8ffc9c518437bcbaf72a4e8ec55c5462ae
|
|
BLAKE2b-256 checksum How to use checksums |
a2703b2f8c409f3ca32593e7be271a9f69ed17e8f019958b9370c12200cbafdd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp311-cp311-win_amd64.whl
| Download URL | flatterer-0.25.0-cp311-cp311-win_amd64.whl |
|---|---|
| Size | 14.0 MB |
| Tags | CPython 3.11 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
defb93ef50246617179dfdee983b4ff3278e9f9605c6b384e9fd960e0cdf182b
|
|
BLAKE2b-256 checksum How to use checksums |
7488c259a9f74bf588b5628dd1d3229fc8046ecb62edcb626183e1df62312beb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp311-cp311-manylinux_2_28_x86_64.whl
| Download URL | flatterer-0.25.0-cp311-cp311-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 14.9 MB |
| Tags | CPython 3.11 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
494a7ed1dc0072a5317909159bb8a4a490a9af27cb310e7e89c6915864217c21
|
|
BLAKE2b-256 checksum How to use checksums |
05db37ce166ab32ec037a342e17479df8886922b18fdc0e887fd8e4218418a05
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp311-cp311-macosx_11_0_arm64.whl
| Download URL | flatterer-0.25.0-cp311-cp311-macosx_11_0_arm64.whl |
|---|---|
| Size | 13.2 MB |
| Tags | CPython 3.11 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
dd588da673b8323ccb491c75cf3b19b91197a70d86a12ce8c108a111b4c5b82a
|
|
BLAKE2b-256 checksum How to use checksums |
8dc3339d804afc8ffc69ce658ec50a8fdcf13614c3c63201395b973c40e42c94
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp310-cp310-win_amd64.whl
| Download URL | flatterer-0.25.0-cp310-cp310-win_amd64.whl |
|---|---|
| Size | 14.0 MB |
| Tags | CPython 3.10 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
eb734a3b0b4499a0d12fab808fdcee196a9abc2585afaa5e48d62d8615368f20
|
|
BLAKE2b-256 checksum How to use checksums |
023bf003f106687e346fef1537f0433e3adf11e906fb890713ac9b4ca638661c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp310-cp310-manylinux_2_28_x86_64.whl
| Download URL | flatterer-0.25.0-cp310-cp310-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 14.9 MB |
| Tags | CPython 3.10 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
6efa82f83134b6bca6950605ac15d18950ce1eb6527bcbd71b29fc5a792ba0b1
|
|
BLAKE2b-256 checksum How to use checksums |
e12a091452c3599d87793ed2e6abee9c89bc2742187490ae401a3670b8e88950
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Release files / flatterer-0.25.0-cp39-cp39-manylinux_2_28_x86_64.whl
| Download URL | flatterer-0.25.0-cp39-cp39-manylinux_2_28_x86_64.whl |
|---|---|
| Size | 14.9 MB |
| Tags | CPython 3.9 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
72cdde00cbc88df410907ecaf44619cb86df960bed1861c6dfdc1050bd2e089a
|
|
BLAKE2b-256 checksum How to use checksums |
590d00455d01b3056e6ca5c89a5740f2859c75bb389ec0a530df22c10e263f9b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.14.1
|
Security for flatterer 0.25.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
