ETL table converter and creator
The Python Toolkit for converting table from one database to another
Databases types
Supported databases. You have to use short name of database short_name:type of database
- ch:clickhouse
- pg:postgresql
- mysql:mysql
- vertica:vertica
- exasol:exasol
Supported database conversions
- mysql to pg
- mysql to vertica
- mysql to exasol
- ch to vertica
- pg to vertica
Credentials
sql_credentials = {
"pg": {
"database": os.getenv("PG_DATABASE"),
"schema": os.getenv("PG_SCHEMA"),
"user": os.getenv("PG_USER"),
"host": os.getenv("PG_HOST"),
"port": os.getenv("PG_PORT"),
"password": os.getenv("PG_PASSWORD"),
#additional params bellow (optional)
"executemany_mode": "values",
"executemany_values_page_size": 10000,
"executemany_batch_page_size": 500,
},
"ch": {
"database": os.getenv("CH_DATABASE"),
"user": os.getenv("CH_USER"),
"host": os.getenv("CH_HOST"),
"port": os.getenv("CH_PORT"),
"password": os.getenv("CH_PASSWORD"),
#additional params bellow (optional)
"connect_args": {
"alt_hosts": "{},{}:{}".format(
os.getenv("CH_HOST_B"), os.getenv("CH_HOST_C"), os.getenv("CH_PORT")
)
},
},
"vertica": {
"database": os.getenv("VERTICA_DATABASE"),
"schema": os.getenv("VERTICA_SCHEMA"),
"user": os.getenv("VERTICA_USER"),
"host": os.getenv("VERTICA_HOST"),
"port": os.getenv("VERTICA_PORT"),
"password": os.getenv("VERTICA_PASSWORD"),
#additional params bellow (optional)
"connect_args": {
"connection_load_balance": True,
"backup_server_node": json.loads(os.getenv("VERTICA_CONFIGS"))[
"backup_server_node"
],
},
},
"mysql": {
"database": os.getenv("MYSQL_DATABASE"),
"user": os.getenv("MYSQL_USER"),
"host": os.getenv("MYSQL_HOST"),
"port": os.getenv("MYSQL_PORT"),
"password": os.getenv("MYSQL_PASSWORD"),
},
}
Usage
pip3 install etl-helper-talenttech
import os
from converter.fields_converter import FieldsConverter
from_db = "mysql"
to_db = "vertica"
tables = ["users", "user_to_mc"]
converter = FieldsConverter(sql_credentials, from_db, to_db, tables=tables, advanced_features=[0, 1])
res = converter.create_list_of_tables(tables, to_create=True, dir_ddl=None)
fields_from = converter.db_worker_from.get_columns(tables[0])
fields_to = converter.db_worker_to.get_columns(tables[0])
print(fields_from)
print(fields_to)
1 maintainer
Metadata
1 maintainer
Release files for etl-helper-talenttech 1.7.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| etl-helper-talenttech-1.7.1.tar.gz | 13.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| etl_helper_talenttech-1.7.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 29.7 kB
Release files / etl-helper-talenttech-1.7.1.tar.gz
| Download URL | etl-helper-talenttech-1.7.1.tar.gz |
|---|---|
| Size | 13.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
410eff34e16ec2db54b681d0e5b6f3e97d780e7be1616faee287beb492897399
|
|
BLAKE2b-256 checksum How to use checksums |
97cee2c84f5d7ed03c0841e7d0a9bc2db40afe386c92cba9fa3d1d8d5d2097e8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/2.0.0 pkginfo/1.9.6 requests/2.31.0 setuptools/69.0.2 requests-toolbelt/1.0.0 tqdm/4.66.1 CPython/3.8.10
|
Release files / etl_helper_talenttech-1.7.1-py3-none-any.whl
| Download URL | etl_helper_talenttech-1.7.1-py3-none-any.whl |
|---|---|
| Size | 16.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
78296cc5d12020f6552a66b41c8cc32eaddcf38faa38c8a2377867f7bac3802a
|
|
BLAKE2b-256 checksum How to use checksums |
e068bdd11976d6c1ae2d0740fa0d67fe8a5f17d837f3d9c4fbb03a73a718ca0f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/2.0.0 pkginfo/1.9.6 requests/2.31.0 setuptools/69.0.2 requests-toolbelt/1.0.0 tqdm/4.66.1 CPython/3.8.10
|
