Multiple tools and utilities for ETL pipelines and others.
Project description
Ditat ETL
Multiple tools and utilities for ETL pipelines and others.
Utils
time_it
Decorator to time function and class method. Additional text can be added.
from ditat_etl.utils import time_it
@time_it()
def f():
'''Do something'''
f()
f time: 0.1
Databases
Useful wrappers for databases and methods to execute queries.
Postgres
It is compatible with pandas.DataFrame interaction, either reading as dataframes and pushing to the db.
from ditat_etl.databases import Postgres
config = {
"database": "xxxx",
"user": "xxxx",
"password": "xxxx",
"host": "xxxxx",
"port": "xxxx"
}
p = Postgres(config)
The main base function is query.
p.query(
query_statement: list or str,
df: bool=False,
as_dict: bool=False,
commit: bool=True,
returning: bool=True,
mogrify: bool=False,
mogrify_tuple: tuple or list=None,
verbose=False
)
This function is a workaround of pandas.to_sql() which drops the table before inserting. It really works like an upsert and it gives you the option to do nothing or update on the column(s) constraint.
p.insert_df_to_sql(
df: pd.DataFrame,
tablename: str,
commit=True,
conflict_on: list=None,
do_update_columns: bool or list=False,
verbose=False
):
This one is similar, it lets you "upsert" without necessarily having a primary key or constraint. Ideally use the previous method.
p.update_df_to_sql(
df: pd.DataFrame,
tablename: str,
on_columns: str or list,
insert_new=True,
commit=True,
verbose=False
):
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for ditat_etl-0.0.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9882c04c6282f4c6122d72fbe7e2ffdd4cc0322684bb10e2cd50dfd18eeeedd7 |
|
MD5 | f9a5ab17bdca708e070b3213c0e8e376 |
|
BLAKE2b-256 | 85ba4694ba0340a5c898f75241f2713bc197a011fc481194e3ce11c39236c2e0 |