Skip to main content

TopMind ETL helper

Project description

ETL table converter and creator

The Python Toolkit for converting table from one database to another

Databases types

Supported databases. You have to use short name of database short_name:type of database

  • ch:clickhouse
  • pg:postgresql
  • mysql:mysql
  • vertica:vertica
  • exasol:exasol

Supported database conversions

  • mysql to pg
  • mysql to vertica
  • mysql to exasol
  • ch to vertica
  • pg to vertica

Credentials

sql_credentials = {
    "pg": {
        "database": os.getenv("PG_DATABASE"),
        "schema": os.getenv("PG_SCHEMA"),
        "user": os.getenv("PG_USER"),
        "host": os.getenv("PG_HOST"),
        "port": os.getenv("PG_PORT"),
        "password": os.getenv("PG_PASSWORD"),
        #additional params bellow (optional) 
        "executemany_mode": "values",
        "executemany_values_page_size": 10000,
        "executemany_batch_page_size": 500,
    },
    "ch": {
        "database": os.getenv("CH_DATABASE"),
        "user": os.getenv("CH_USER"),
        "host": os.getenv("CH_HOST"),
        "port": os.getenv("CH_PORT"),
        "password": os.getenv("CH_PASSWORD"),
        #additional params bellow (optional) 
        "connect_args": {
            "alt_hosts": "{},{}:{}".format(
                os.getenv("CH_HOST_B"), os.getenv("CH_HOST_C"), os.getenv("CH_PORT")
            )
        },
    },
    "vertica": {
        "database": os.getenv("VERTICA_DATABASE"),
        "schema": os.getenv("VERTICA_SCHEMA"),
        "user": os.getenv("VERTICA_USER"),
        "host": os.getenv("VERTICA_HOST"),
        "port": os.getenv("VERTICA_PORT"),
        "password": os.getenv("VERTICA_PASSWORD"),
        #additional params bellow (optional) 
        "connect_args": {
            "connection_load_balance": True,
            "backup_server_node": json.loads(os.getenv("VERTICA_CONFIGS"))[
                "backup_server_node"
            ],
        },
    },
    "mysql": {
        "database": os.getenv("MYSQL_DATABASE"),
        "user": os.getenv("MYSQL_USER"),
        "host": os.getenv("MYSQL_HOST"),
        "port": os.getenv("MYSQL_PORT"),
        "password": os.getenv("MYSQL_PASSWORD"),
    },
}

Usage

pip3 install etl-helper-talenttech
import os
from converter.fields_converter import FieldsConverter


from_db = "mysql"
to_db = "vertica"
  
tables = ["users", "user_to_mc"]
converter = FieldsConverter(sql_credentials, from_db, to_db, tables=tables, advanced_features=[0, 1])
res = converter.create_list_of_tables(tables, to_create=True, dir_ddl=None)
fields_from = converter.db_worker_from.get_columns(tables[0])
fields_to = converter.db_worker_to.get_columns(tables[0])
print(fields_from)
print(fields_to)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

etl-helper-talenttech-1.7.1.tar.gz (13.3 kB view details)

Uploaded Source

Built Distribution

etl_helper_talenttech-1.7.1-py3-none-any.whl (16.4 kB view details)

Uploaded Python 3

File details

Details for the file etl-helper-talenttech-1.7.1.tar.gz.

File metadata

  • Download URL: etl-helper-talenttech-1.7.1.tar.gz
  • Upload date:
  • Size: 13.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.9.6 requests/2.31.0 setuptools/69.0.2 requests-toolbelt/1.0.0 tqdm/4.66.1 CPython/3.8.10

File hashes

Hashes for etl-helper-talenttech-1.7.1.tar.gz
Algorithm Hash digest
SHA256 410eff34e16ec2db54b681d0e5b6f3e97d780e7be1616faee287beb492897399
MD5 6a8e6c96bfd053b06b4c96e5aac86aa3
BLAKE2b-256 97cee2c84f5d7ed03c0841e7d0a9bc2db40afe386c92cba9fa3d1d8d5d2097e8

See more details on using hashes here.

File details

Details for the file etl_helper_talenttech-1.7.1-py3-none-any.whl.

File metadata

  • Download URL: etl_helper_talenttech-1.7.1-py3-none-any.whl
  • Upload date:
  • Size: 16.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.9.6 requests/2.31.0 setuptools/69.0.2 requests-toolbelt/1.0.0 tqdm/4.66.1 CPython/3.8.10

File hashes

Hashes for etl_helper_talenttech-1.7.1-py3-none-any.whl
Algorithm Hash digest
SHA256 78296cc5d12020f6552a66b41c8cc32eaddcf38faa38c8a2377867f7bac3802a
MD5 2083867e9187ed0c1aa136f6c240523c
BLAKE2b-256 e068bdd11976d6c1ae2d0740fa0d67fe8a5f17d837f3d9c4fbb03a73a718ca0f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page