Skip to main content

Convert your csvs fast

Project description

CSVS Convert

Converts CSV files into XLSX/SQLITE/POSTGRESQL/PARQUET fast.

Install

pip install csvs_convert

Docs

Full Documentaion

Aims

  • Thorough type guessing of CSV columns, so there is no need to configure types of each field. Scans whole file first to make sure all types in a column are consistent. Can detect over 30 date/time formats as well as JSON data.
  • Quick conversions/type guessing (uses rust underneath). Uses fast methods specific for each output format:
    • copy for postgres
    • Prepared statements for sqlite using c API.
    • Arrow reader for parquet
    • Write only mode for libxlsxwriter
  • Tries to limit errors when inserting data into database by resorting to "text" if type guessing can't determine a more specific type.
  • When inserting into existing databases automatically migrate schema of target to allow for new data (evolve option).
  • Memory efficient. All csvs and outputs are streamed so all conversions should take up very little memory.
  • Gather stats and information about CSV files into datapacakge.json file which can use it for customizing conversion.

Drawbacks

  • CSV files currently need header rows.
  • Whole file needs to be on disk as whole CSV is analyzed therefore files are read twice.

Conversion Docs

This is the python library, providing bindings to the rust library.

Contribute on github

Usage From CSV files.

import csvs_convert

#sqlite
csvs_convert.csvs_to_sqlite("sqlite.db", ["file.csv"])
#postgres
csvs_convert.csvs_to_postgres("postgresql://user:postgres@localhost/db", ["file.csv"])
#parquet
csvs_convert.csvs_to_parquet("output", ["file.csv"])
#xlsx
csvs_convert.csvs_to_xlsx("output.xlsx", ["sqlite.db"])

Usage from datapackage

A datapackage is a file that contains metadata about the tables its specification is described here.

To generate datapackage.json file you can use:

csvs_convert.csvs_to_datapackage('path/to/datapackage.json', ["fixtures/large/csv/data.csv"])

Other tools can also generate these files.

You can use this file and alter it as needed. Mostly it is useful if you want to use the same schema across multiple files, as it will save time not having to do the type guessing for every file.

When referring to a datapackage you can either reference:

  • A datapackage.json file.
  • A datapackage directory containing a datapackage.json file. e.g. /a/datapackage/dir
  • A zip file containing a datapackage.json file. e.g. my_datapackage.zip

Examples:

import csvs_convert

#sqlite
csvs_convert.datapackage_to_sqlite("sqlite.db", "path/to/datapackage.json")
#postgres
csvs_convert.datapackage_to_postgres("postgresql://user:postgres@localhost/db", "path/to/datapackage.json")
#parquet
csvs_convert.datapackage_to_parquet("path/to/directory", ["sqlite.db"])
#xlsx
csvs_convert.datapackage_to_xlsx("output.xlsx", "path/to/datapackage.json")

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

csvs_convert-0.7.3.tar.gz (99.2 kB view details)

Uploaded Source

Built Distributions

csvs_convert-0.7.3-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.5 MB view details)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

csvs_convert-0.7.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.5 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

csvs_convert-0.7.3-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (10.0 MB view details)

Uploaded CPython 3.11 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

csvs_convert-0.7.3-cp311-cp311-macosx_10_7_x86_64.whl (5.2 MB view details)

Uploaded CPython 3.11 macOS 10.7+ x86-64

csvs_convert-0.7.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.5 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

csvs_convert-0.7.3-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (10.0 MB view details)

Uploaded CPython 3.10 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

csvs_convert-0.7.3-cp310-cp310-macosx_10_7_x86_64.whl (5.2 MB view details)

Uploaded CPython 3.10 macOS 10.7+ x86-64

csvs_convert-0.7.3-cp39-none-win_amd64.whl (4.4 MB view details)

Uploaded CPython 3.9 Windows x86-64

csvs_convert-0.7.3-cp39-none-win32.whl (4.1 MB view details)

Uploaded CPython 3.9 Windows x86

csvs_convert-0.7.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.5 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

csvs_convert-0.7.3-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (10.0 MB view details)

Uploaded CPython 3.9 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

csvs_convert-0.7.3-cp39-cp39-macosx_10_7_x86_64.whl (5.2 MB view details)

Uploaded CPython 3.9 macOS 10.7+ x86-64

csvs_convert-0.7.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.5 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

csvs_convert-0.7.3-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (10.0 MB view details)

Uploaded CPython 3.8 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

csvs_convert-0.7.3-cp38-cp38-macosx_10_7_x86_64.whl (5.2 MB view details)

Uploaded CPython 3.8 macOS 10.7+ x86-64

csvs_convert-0.7.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.5 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

csvs_convert-0.7.3-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (10.0 MB view details)

Uploaded CPython 3.7m macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

csvs_convert-0.7.3-cp37-cp37m-macosx_10_7_x86_64.whl (5.2 MB view details)

Uploaded CPython 3.7m macOS 10.7+ x86-64

File details

Details for the file csvs_convert-0.7.3.tar.gz.

File metadata

  • Download URL: csvs_convert-0.7.3.tar.gz
  • Upload date:
  • Size: 99.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.15

File hashes

Hashes for csvs_convert-0.7.3.tar.gz
Algorithm Hash digest
SHA256 46d47badea554ddd0ae1cd3f524d5eff3df46e5a2e957d600e7399cf4cb67cd6
MD5 652cf405b1008faa398edc40b4b30a92
BLAKE2b-256 73e1825491faf1acc5851360d2a6a888ca75a0b4451b4d1ba1ed8c34a02509fe

See more details on using hashes here.

File details

Details for the file csvs_convert-0.7.3-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for csvs_convert-0.7.3-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 be5e4f4e225504140cceae84c969d8d167f3aa3a237d60046d4a612574c859ab
MD5 aa36f31c24ebfde9a18a9907c5be5112
BLAKE2b-256 007303884b7cccb780b4d60cc280950f95219e18ff86f46035512889af83ce55

See more details on using hashes here.

File details

Details for the file csvs_convert-0.7.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for csvs_convert-0.7.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 84baca3eb3435f4dc33656914a276d9d13a364fa1ad26da7a2f58122a9b8789c
MD5 5dd0ffe79f6fd4a7f5638ef6800b33dc
BLAKE2b-256 65a314876cf0e0360c6a525ad8ef56e6c7fa0faff5e34dcde50f62e731e52fd9

See more details on using hashes here.

File details

Details for the file csvs_convert-0.7.3-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for csvs_convert-0.7.3-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 6f3b304e1b1b56d767b3a4aa742ee9c7b2f1aaa2f8a31279a9340a82d2514b9c
MD5 2fbf5e60f87b3ff53083b3857b72e1ca
BLAKE2b-256 0f73352ad31e2c5dde9c55f9c74d9dcd4cf9aef43d2c6f3257b335ed17833e9b

See more details on using hashes here.

File details

Details for the file csvs_convert-0.7.3-cp311-cp311-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for csvs_convert-0.7.3-cp311-cp311-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 4f77180c3a7e3be8beda86d5e27eca37e1f140edd6eaa73c04373a5933dc4a96
MD5 0b586b0677e5464a1846a422e47fc0a8
BLAKE2b-256 1cdceedcd10c3243c7449cd3450e81abc85c8a4f658f859a174d8b8ec9d82529

See more details on using hashes here.

File details

Details for the file csvs_convert-0.7.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for csvs_convert-0.7.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 690740845b01c7c2d020a732025fb05bf2163730d985eb7e0ba2e308666ce162
MD5 e2e6e3a48a188161b7270b70edb6d0b8
BLAKE2b-256 e568a655644351488233cf8eaf13518d879aa2b05e76b27ec424a2640f3ffcf2

See more details on using hashes here.

File details

Details for the file csvs_convert-0.7.3-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for csvs_convert-0.7.3-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 5b2d337d8ce5222e0e5b14cba2119e478f1d5765d7ea57aa079c7a32c11dc643
MD5 a07fb655e9377fe0059fe5fedc02d839
BLAKE2b-256 a8e7c2504a29ac481726d79c9143959a106fae0ee068502e0c43c34b80fe9874

See more details on using hashes here.

File details

Details for the file csvs_convert-0.7.3-cp310-cp310-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for csvs_convert-0.7.3-cp310-cp310-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 7b1967e8c04a6deab6cd6407f746a078f43f443b3c45b7b19b5321bf393d94ce
MD5 24df93d4b4283e031755fccfd4b77ff2
BLAKE2b-256 4d786cdf00d0fe6737c64dbc0b91882dbc071821875739debe23ca5684081472

See more details on using hashes here.

File details

Details for the file csvs_convert-0.7.3-cp39-none-win_amd64.whl.

File metadata

File hashes

Hashes for csvs_convert-0.7.3-cp39-none-win_amd64.whl
Algorithm Hash digest
SHA256 e82e2b0c1d399c0d6a5c9b2fc289b36edeb39feb850ae88d7edee2cdb8932813
MD5 1f7b86cc3bf0cb1ac81d820a000f9664
BLAKE2b-256 a3dbcfbdc5d0cc8c842729f457896574a824e6118c1b39383025af76302fde96

See more details on using hashes here.

File details

Details for the file csvs_convert-0.7.3-cp39-none-win32.whl.

File metadata

  • Download URL: csvs_convert-0.7.3-cp39-none-win32.whl
  • Upload date:
  • Size: 4.1 MB
  • Tags: CPython 3.9, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.15

File hashes

Hashes for csvs_convert-0.7.3-cp39-none-win32.whl
Algorithm Hash digest
SHA256 cca8905798377330ef2444a85276e07736dc29c4b5d6c87c660d2759da7ffd5c
MD5 a13b25f393c7aed09effc094714708eb
BLAKE2b-256 f512bc37ad98d27900a7b809607bddfbb28e099ddcb61b2b8bcf6f58e8d28335

See more details on using hashes here.

File details

Details for the file csvs_convert-0.7.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for csvs_convert-0.7.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 6800f63568727fb570155702cdd4382baaa9ee84042c7390e15fc74ce465176c
MD5 6c46b0880d8d6583e65405d8e5dd90b0
BLAKE2b-256 bd02e6057f64cd7d79c0cbd28b22d41d426701c29e36f4b75e03611e30f601c5

See more details on using hashes here.

File details

Details for the file csvs_convert-0.7.3-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for csvs_convert-0.7.3-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 0777bfcd67a6c91140d88f8b80ecb10c542388dec9d3379c44ef704bc933a2f2
MD5 b94cf9047313f4435a21b870110d8aac
BLAKE2b-256 b363d999e6fd01a74e32cc210685a659e97f965cf5167901c5f795d9597be9a8

See more details on using hashes here.

File details

Details for the file csvs_convert-0.7.3-cp39-cp39-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for csvs_convert-0.7.3-cp39-cp39-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 39c5434d35066d03bc9eb00cb4b939c7a627118f6c390c4a36ba20ef6147f638
MD5 75d5ee14fc36449f4e0dd8668bb01eb0
BLAKE2b-256 061f0eff45a65bc3f3d0f28a36e5926a8d0ab2129b7e68e3444433d5a5bcc606

See more details on using hashes here.

File details

Details for the file csvs_convert-0.7.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for csvs_convert-0.7.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 e5e8829b5f5caba43e0849ca1c9aa1f4c85dc55b5cd62c8ce10efad0a7f05a66
MD5 42f901a2c5163bce70d5fdd171ebe293
BLAKE2b-256 c274ad0391e4c61f73b7c78abe8c70a4343ba815edb408a59014b70ee9afbc7b

See more details on using hashes here.

File details

Details for the file csvs_convert-0.7.3-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for csvs_convert-0.7.3-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 2111584edf239a4fee5ddf8c7ae5930b44b0be50dcd921d5a4abea8851b4e755
MD5 1ab2d7f23b277ee30f383fdc7799179a
BLAKE2b-256 019e211e7116fc618ce92663a1d83760c37cf7e9d42c85ea51277d7f79ccfdbc

See more details on using hashes here.

File details

Details for the file csvs_convert-0.7.3-cp38-cp38-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for csvs_convert-0.7.3-cp38-cp38-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 5b774aa1510c42f081cf7b8719f5a68680e9159fa82408f91aeb1601d5b131d3
MD5 87ef3beab40d4009ca86ca0baa3e8edc
BLAKE2b-256 815a3a70099f4e41de0e167dfc0d9d3a71e49f4eb79ac98bdd6e44480d083252

See more details on using hashes here.

File details

Details for the file csvs_convert-0.7.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for csvs_convert-0.7.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 92955b4ee922996f88df1a5b782bd980f4d9f26c642c3b084e8d90c3278e62df
MD5 89a73b2657c39780d164ffffd95fdbe4
BLAKE2b-256 1e37f3c315307beb59405ca76cec0d7ec858221e92b9894765b0e75af1b8ce0e

See more details on using hashes here.

File details

Details for the file csvs_convert-0.7.3-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for csvs_convert-0.7.3-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 f9fca69346ca5dfb46ae86ca7ef544487a666cea79405122aab4d06a0931ebe1
MD5 83029ad120816512f28cd456bebfe1c2
BLAKE2b-256 94d64db3760916ddece74b9446655f103f0f81f86b18e5bdbf713802b7732362

See more details on using hashes here.

File details

Details for the file csvs_convert-0.7.3-cp37-cp37m-macosx_10_7_x86_64.whl.

File metadata

File hashes

Hashes for csvs_convert-0.7.3-cp37-cp37m-macosx_10_7_x86_64.whl
Algorithm Hash digest
SHA256 0faeac81340e04cd591a8c7ae9e7a6349151951ed7684d9338c267c9299862e9
MD5 835b78b834873cb7d33b2fda2124efe2
BLAKE2b-256 771fd8c6d060d0aaad070114101d3fbb642e659e5d99119982de5facad87e3c4

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page