ETL framework for Colosscious Pharmquer
Project description
lichens
ETL framework for Colosscious Pharmquer.
It contains a Web UI and a backend. See Colosscious Official Website for more.
Installation
pip install lichens
Example
Setup
# Command
python -m lichens --help
# Output:
# Usage: python -m lichens [OPTIONS] COMMAND [ARGS]...
#
# Options:
# --help Show this message and exit.
#
# Commands:
# add-etl Add an ETL program setting from a JSON
# get-template Generate an ETL program setting to a JSON
# migrate Make and migrate the system tables.
# Initialize the database if needed.
python -m lichens migrate \
-c driver://user:pass@localhost:port/dbname
# Create an ETL setting template
python -m lichens get-template \
# Create it in the database
python -m lichens add-etl \
--connection-string driver://user:pass@localhost:port/dbname \
--json-config etl-setting.json
Write an ETL
See full example
# Create an ETL Manager
from lichens import EtlManager
em = EtlManager(constr=CONNECTION_STRING, name=ETL_NAME)
# Use Pandera to validate data
from lichens import DataFrameSchema, check_io
schema = DataFrameSchema(...)
validated_df = schema(df)
## Load df to datebase
em.load_df(
df=df,
tablename="sample_table",
schema="public",
if_exists="raise_error",
chunksize=500,
unique_key=["column1", "column2"],
)
# Update log and archive file
em.update_status(
filename=f,
status=_status,
last_log=_last_log)
em.move(
src=os.path.join(em.src_folder, f),
status=_status)
# Use state highlighted logger
from lichens.logging import logger as log
# Run as scheduled
## Define a function to be executed
def your_function():
# Your function logic here
pass
# Run the function every minute for a total of 5 times
em.run_as_schtask(your_function, '*/1 * * * *', times_=5)
## Decorate the function with the scheduled decorator
@em.scheduled(crontab='*/1 * * * *', times_=5)
def your_scheduled_function():
# Your function logic here
pass
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
lichens-0.1.0.tar.gz
(24.7 kB
view hashes)
Built Distribution
lichens-0.1.0-py3-none-any.whl
(17.9 kB
view hashes)