TopMind ETL helper
Project description
ETL table converter and creator
The Python Toolkit for converting table from one database to another
Databases types
Supported databases. You have to use short name of database short_name:type of database
- ch:clickhouse
- pg:postgresql
- mysql:mysql
- vertica:vertica
- exasol:exasol
Supported database conversions
- mysql to pg
- mysql to vertica
- mysql to exasol
- ch to vertica
- pg to vertica
Credentials
sql_credentials = {
"pg": {
"database": os.getenv("PG_DATABASE"),
"schema": os.getenv("PG_SCHEMA"),
"user": os.getenv("PG_USER"),
"host": os.getenv("PG_HOST"),
"port": os.getenv("PG_PORT"),
"password": os.getenv("PG_PASSWORD"),
#additional params bellow (optional)
"executemany_mode": "values",
"executemany_values_page_size": 10000,
"executemany_batch_page_size": 500,
},
"ch": {
"database": os.getenv("CH_DATABASE"),
"user": os.getenv("CH_USER"),
"host": os.getenv("CH_HOST"),
"port": os.getenv("CH_PORT"),
"password": os.getenv("CH_PASSWORD"),
#additional params bellow (optional)
"connect_args": {
"alt_hosts": "{},{}:{}".format(
os.getenv("CH_HOST_B"), os.getenv("CH_HOST_C"), os.getenv("CH_PORT")
)
},
},
"vertica": {
"database": os.getenv("VERTICA_DATABASE"),
"schema": os.getenv("VERTICA_SCHEMA"),
"user": os.getenv("VERTICA_USER"),
"host": os.getenv("VERTICA_HOST"),
"port": os.getenv("VERTICA_PORT"),
"password": os.getenv("VERTICA_PASSWORD"),
#additional params bellow (optional)
"connect_args": {
"connection_load_balance": True,
"backup_server_node": json.loads(os.getenv("VERTICA_CONFIGS"))[
"backup_server_node"
],
},
},
"mysql": {
"database": os.getenv("MYSQL_DATABASE"),
"user": os.getenv("MYSQL_USER"),
"host": os.getenv("MYSQL_HOST"),
"port": os.getenv("MYSQL_PORT"),
"password": os.getenv("MYSQL_PASSWORD"),
},
}
Usage
pip3 install etl-helper-talenttech
import os
from converter.fields_converter import FieldsConverter
from_db = "mysql"
to_db = "vertica"
tables = ["users", "user_to_mc"]
converter = FieldsConverter(sql_credentials, from_db, to_db, tables=tables, advanced_features=[0, 1])
res = converter.create_list_of_tables(tables, to_create=True, dir_ddl=None)
fields_from = converter.db_worker_from.get_columns(tables[0])
fields_to = converter.db_worker_to.get_columns(tables[0])
print(fields_from)
print(fields_to)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file etl-helper-talenttech-1.7.1.tar.gz
.
File metadata
- Download URL: etl-helper-talenttech-1.7.1.tar.gz
- Upload date:
- Size: 13.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/2.0.0 pkginfo/1.9.6 requests/2.31.0 setuptools/69.0.2 requests-toolbelt/1.0.0 tqdm/4.66.1 CPython/3.8.10
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 410eff34e16ec2db54b681d0e5b6f3e97d780e7be1616faee287beb492897399 |
|
MD5 | 6a8e6c96bfd053b06b4c96e5aac86aa3 |
|
BLAKE2b-256 | 97cee2c84f5d7ed03c0841e7d0a9bc2db40afe386c92cba9fa3d1d8d5d2097e8 |
File details
Details for the file etl_helper_talenttech-1.7.1-py3-none-any.whl
.
File metadata
- Download URL: etl_helper_talenttech-1.7.1-py3-none-any.whl
- Upload date:
- Size: 16.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/2.0.0 pkginfo/1.9.6 requests/2.31.0 setuptools/69.0.2 requests-toolbelt/1.0.0 tqdm/4.66.1 CPython/3.8.10
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 78296cc5d12020f6552a66b41c8cc32eaddcf38faa38c8a2377867f7bac3802a |
|
MD5 | 2083867e9187ed0c1aa136f6c240523c |
|
BLAKE2b-256 | e068bdd11976d6c1ae2d0740fa0d67fe8a5f17d837f3d9c4fbb03a73a718ca0f |