Convert your csvs fast
Project description
CSVS Convert
Converts CSV files into XLSX/SQLITE/POSTGRESQL/PARQUET fast.
Install
pip install csvs_convert
Docs
Aims
- Thorough type guessing of CSV columns, so there is no need to configure types of each field. Scans whole file first to make sure all types in a column are consistent. Can detect over 30 date/time formats as well as JSON data.
- Quick conversions/type guessing (uses rust underneath). Uses fast methods specific for each output format:
copy
for postgres- Prepared statements for sqlite using c API.
- Arrow reader for parquet
- Write only mode for libxlsxwriter
- Tries to limit errors when inserting data into database by resorting to "text" if type guessing can't determine a more specific type.
- When inserting into existing databases automatically migrate schema of target to allow for new data (
evolve
option). - Memory efficient. All csvs and outputs are streamed so all conversions should take up very little memory.
- Gather stats and information about CSV files into datapacakge.json file which can use it for customizing conversion.
Drawbacks
- CSV files currently need header rows.
- Whole file needs to be on disk as whole CSV is analyzed therefore files are read twice.
Conversion Docs
This is the python library, providing bindings to the rust library.
Usage From CSV files.
import csvs_convert
#sqlite
csvs_convert.csvs_to_sqlite("sqlite.db", ["file.csv"])
#postgres
csvs_convert.csvs_to_postgres("postgresql://user:postgres@localhost/db", ["file.csv"])
#parquet
csvs_convert.csvs_to_parquet("output", ["file.csv"])
#xlsx
csvs_convert.csvs_to_xlsx("output.xlsx", ["sqlite.db"])
Usage from datapackage
A datapackage is a file that contains metadata about the tables its specification is described here.
To generate datapackage.json
file you can use:
csvs_convert.csvs_to_datapackage('path/to/datapackage.json', ["fixtures/large/csv/data.csv"])
Other tools can also generate these files.
You can use this file and alter it as needed. Mostly it is useful if you want to use the same schema across multiple files, as it will save time not having to do the type guessing for every file.
When referring to a datapackage you can either reference:
- A
datapackage.json
file. - A datapackage directory containing a
datapackage.json
file. e.g./a/datapackage/dir
- A zip file containing a
datapackage.json
file. e.g.my_datapackage.zip
Examples:
import csvs_convert
#sqlite
csvs_convert.datapackage_to_sqlite("sqlite.db", "path/to/datapackage.json")
#postgres
csvs_convert.datapackage_to_postgres("postgresql://user:postgres@localhost/db", "path/to/datapackage.json")
#parquet
csvs_convert.datapackage_to_parquet("path/to/directory", ["sqlite.db"])
#xlsx
csvs_convert.datapackage_to_xlsx("output.xlsx", "path/to/datapackage.json")
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
csvs_convert-0.7.3.tar.gz
(99.2 kB
view hashes)
Built Distributions
Close
Hashes for csvs_convert-0.7.3-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | be5e4f4e225504140cceae84c969d8d167f3aa3a237d60046d4a612574c859ab |
|
MD5 | aa36f31c24ebfde9a18a9907c5be5112 |
|
BLAKE2b-256 | 007303884b7cccb780b4d60cc280950f95219e18ff86f46035512889af83ce55 |
Close
Hashes for csvs_convert-0.7.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 84baca3eb3435f4dc33656914a276d9d13a364fa1ad26da7a2f58122a9b8789c |
|
MD5 | 5dd0ffe79f6fd4a7f5638ef6800b33dc |
|
BLAKE2b-256 | 65a314876cf0e0360c6a525ad8ef56e6c7fa0faff5e34dcde50f62e731e52fd9 |
Close
Hashes for csvs_convert-0.7.3-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6f3b304e1b1b56d767b3a4aa742ee9c7b2f1aaa2f8a31279a9340a82d2514b9c |
|
MD5 | 2fbf5e60f87b3ff53083b3857b72e1ca |
|
BLAKE2b-256 | 0f73352ad31e2c5dde9c55f9c74d9dcd4cf9aef43d2c6f3257b335ed17833e9b |
Close
Hashes for csvs_convert-0.7.3-cp311-cp311-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4f77180c3a7e3be8beda86d5e27eca37e1f140edd6eaa73c04373a5933dc4a96 |
|
MD5 | 0b586b0677e5464a1846a422e47fc0a8 |
|
BLAKE2b-256 | 1cdceedcd10c3243c7449cd3450e81abc85c8a4f658f859a174d8b8ec9d82529 |
Close
Hashes for csvs_convert-0.7.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 690740845b01c7c2d020a732025fb05bf2163730d985eb7e0ba2e308666ce162 |
|
MD5 | e2e6e3a48a188161b7270b70edb6d0b8 |
|
BLAKE2b-256 | e568a655644351488233cf8eaf13518d879aa2b05e76b27ec424a2640f3ffcf2 |
Close
Hashes for csvs_convert-0.7.3-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5b2d337d8ce5222e0e5b14cba2119e478f1d5765d7ea57aa079c7a32c11dc643 |
|
MD5 | a07fb655e9377fe0059fe5fedc02d839 |
|
BLAKE2b-256 | a8e7c2504a29ac481726d79c9143959a106fae0ee068502e0c43c34b80fe9874 |
Close
Hashes for csvs_convert-0.7.3-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7b1967e8c04a6deab6cd6407f746a078f43f443b3c45b7b19b5321bf393d94ce |
|
MD5 | 24df93d4b4283e031755fccfd4b77ff2 |
|
BLAKE2b-256 | 4d786cdf00d0fe6737c64dbc0b91882dbc071821875739debe23ca5684081472 |
Close
Hashes for csvs_convert-0.7.3-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e82e2b0c1d399c0d6a5c9b2fc289b36edeb39feb850ae88d7edee2cdb8932813 |
|
MD5 | 1f7b86cc3bf0cb1ac81d820a000f9664 |
|
BLAKE2b-256 | a3dbcfbdc5d0cc8c842729f457896574a824e6118c1b39383025af76302fde96 |
Close
Hashes for csvs_convert-0.7.3-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cca8905798377330ef2444a85276e07736dc29c4b5d6c87c660d2759da7ffd5c |
|
MD5 | a13b25f393c7aed09effc094714708eb |
|
BLAKE2b-256 | f512bc37ad98d27900a7b809607bddfbb28e099ddcb61b2b8bcf6f58e8d28335 |
Close
Hashes for csvs_convert-0.7.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6800f63568727fb570155702cdd4382baaa9ee84042c7390e15fc74ce465176c |
|
MD5 | 6c46b0880d8d6583e65405d8e5dd90b0 |
|
BLAKE2b-256 | bd02e6057f64cd7d79c0cbd28b22d41d426701c29e36f4b75e03611e30f601c5 |
Close
Hashes for csvs_convert-0.7.3-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0777bfcd67a6c91140d88f8b80ecb10c542388dec9d3379c44ef704bc933a2f2 |
|
MD5 | b94cf9047313f4435a21b870110d8aac |
|
BLAKE2b-256 | b363d999e6fd01a74e32cc210685a659e97f965cf5167901c5f795d9597be9a8 |
Close
Hashes for csvs_convert-0.7.3-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 39c5434d35066d03bc9eb00cb4b939c7a627118f6c390c4a36ba20ef6147f638 |
|
MD5 | 75d5ee14fc36449f4e0dd8668bb01eb0 |
|
BLAKE2b-256 | 061f0eff45a65bc3f3d0f28a36e5926a8d0ab2129b7e68e3444433d5a5bcc606 |
Close
Hashes for csvs_convert-0.7.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e5e8829b5f5caba43e0849ca1c9aa1f4c85dc55b5cd62c8ce10efad0a7f05a66 |
|
MD5 | 42f901a2c5163bce70d5fdd171ebe293 |
|
BLAKE2b-256 | c274ad0391e4c61f73b7c78abe8c70a4343ba815edb408a59014b70ee9afbc7b |
Close
Hashes for csvs_convert-0.7.3-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2111584edf239a4fee5ddf8c7ae5930b44b0be50dcd921d5a4abea8851b4e755 |
|
MD5 | 1ab2d7f23b277ee30f383fdc7799179a |
|
BLAKE2b-256 | 019e211e7116fc618ce92663a1d83760c37cf7e9d42c85ea51277d7f79ccfdbc |
Close
Hashes for csvs_convert-0.7.3-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5b774aa1510c42f081cf7b8719f5a68680e9159fa82408f91aeb1601d5b131d3 |
|
MD5 | 87ef3beab40d4009ca86ca0baa3e8edc |
|
BLAKE2b-256 | 815a3a70099f4e41de0e167dfc0d9d3a71e49f4eb79ac98bdd6e44480d083252 |
Close
Hashes for csvs_convert-0.7.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 92955b4ee922996f88df1a5b782bd980f4d9f26c642c3b084e8d90c3278e62df |
|
MD5 | 89a73b2657c39780d164ffffd95fdbe4 |
|
BLAKE2b-256 | 1e37f3c315307beb59405ca76cec0d7ec858221e92b9894765b0e75af1b8ce0e |
Close
Hashes for csvs_convert-0.7.3-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f9fca69346ca5dfb46ae86ca7ef544487a666cea79405122aab4d06a0931ebe1 |
|
MD5 | 83029ad120816512f28cd456bebfe1c2 |
|
BLAKE2b-256 | 94d64db3760916ddece74b9446655f103f0f81f86b18e5bdbf713802b7732362 |
Close
Hashes for csvs_convert-0.7.3-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0faeac81340e04cd591a8c7ae9e7a6349151951ed7684d9338c267c9299862e9 |
|
MD5 | 835b78b834873cb7d33b2fda2124efe2 |
|
BLAKE2b-256 | 771fd8c6d060d0aaad070114101d3fbb642e659e5d99119982de5facad87e3c4 |