ETL table converter and creator
The Python Toolkit for converting table from one database to another
Databases types
Supported databases. You have to use short name of database short_name:type of database
- ch:clickhouse
- pg:postgresql
- mysql:mysql
- vertica:vertica
- exasol:exasol
Supported database conversions
- mysql to pg
- mysql to vertica
- mysql to exasol
- ch to vertica
- pg to vertica
Credentials
sql_credentials = {
"pg": {
"database": os.getenv("PG_DATABASE"),
"schema": os.getenv("PG_SCHEMA"),
"user": os.getenv("PG_USER"),
"host": os.getenv("PG_HOST"),
"port": os.getenv("PG_PORT"),
"password": os.getenv("PG_PASSWORD"),
#additional params bellow (optional)
"executemany_mode": "values",
"executemany_values_page_size": 10000,
"executemany_batch_page_size": 500,
},
"ch": {
"database": os.getenv("CH_DATABASE"),
"user": os.getenv("CH_USER"),
"host": os.getenv("CH_HOST"),
"port": os.getenv("CH_PORT"),
"password": os.getenv("CH_PASSWORD"),
#additional params bellow (optional)
"connect_args": {
"alt_hosts": "{},{}:{}".format(
os.getenv("CH_HOST_B"), os.getenv("CH_HOST_C"), os.getenv("CH_PORT")
)
},
},
"vertica": {
"database": os.getenv("VERTICA_DATABASE"),
"schema": os.getenv("VERTICA_SCHEMA"),
"user": os.getenv("VERTICA_USER"),
"host": os.getenv("VERTICA_HOST"),
"port": os.getenv("VERTICA_PORT"),
"password": os.getenv("VERTICA_PASSWORD"),
#additional params bellow (optional)
"connect_args": {
"connection_load_balance": True,
"backup_server_node": json.loads(os.getenv("VERTICA_CONFIGS"))[
"backup_server_node"
],
},
},
"mysql": {
"database": os.getenv("MYSQL_DATABASE"),
"user": os.getenv("MYSQL_USER"),
"host": os.getenv("MYSQL_HOST"),
"port": os.getenv("MYSQL_PORT"),
"password": os.getenv("MYSQL_PASSWORD"),
},
}
Usage
pip3 install etl-converter-talenttech
import os
from converter.fields_converter import FieldsConverter
tables = [
"directions",
"users"
]
from_db = "mysql"
to_db = "pg"
converter = FieldsConverter(sql_credentials, from_db, to_db, tables=tables)
print(converter.get_columns(tables[0]))
print(converter.get_columns(tables[0], table_from=False))
print(converter.create_list_of_tables(tables, to_create=False, dir=None))
1 maintainer
Metadata
1 maintainer
Release files for etl-converter-talenttech 1.2.9
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| etl-converter-talenttech-1.2.9.tar.gz | 7.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| etl_converter_talenttech-1.2.9-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 16.0 kB
Release files / etl-converter-talenttech-1.2.9.tar.gz
| Download URL | etl-converter-talenttech-1.2.9.tar.gz |
|---|---|
| Size | 7.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
40745d92898725d419b9017684050254112a6b2f346d1d63a8f7335d415b8b3d
|
|
BLAKE2b-256 checksum How to use checksums |
677b3f6f2caecffbb7a65796aaa1dac37e6e1b9f3a6c614f78899b7a6b89fb9d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.3.0 requests-toolbelt/0.8.0 tqdm/4.48.2 CPython/3.6.11
|
Release files / etl_converter_talenttech-1.2.9-py3-none-any.whl
| Download URL | etl_converter_talenttech-1.2.9-py3-none-any.whl |
|---|---|
| Size | 8.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
60020bb4505ccead13a2cfd60d63a2c231e97a2b32c41ad305bba822590c9300
|
|
BLAKE2b-256 checksum How to use checksums |
c9665a8c9b8f4185850091f59edde6acded0c17c1d867233a0042cd7f534ce7e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.3.0 requests-toolbelt/0.8.0 tqdm/4.48.2 CPython/3.6.11
|
