lichens 0.1.0

ETL framework for Colosscious Pharmquer

lichens

ETL framework for Colosscious Pharmquer.
It contains a Web UI and a backend. See Colosscious Official Website for more.

Installation

pip install lichens

Example

Setup

# Command
python -m lichens --help
# Output:
# Usage: python -m lichens [OPTIONS] COMMAND [ARGS]...
# 
# Options:
#   --help  Show this message and exit.
# 
# Commands:
#   add-etl       Add an ETL program setting from a JSON
#   get-template  Generate an ETL program setting to a JSON
#   migrate       Make and migrate the system tables.

# Initialize the database if needed. 
python -m lichens migrate \
-c driver://user:pass@localhost:port/dbname 


# Create an ETL setting template
python -m lichens get-template \


# Create it in the database
python -m lichens add-etl \
--connection-string driver://user:pass@localhost:port/dbname \
--json-config etl-setting.json

Write an ETL

See full example

# Create an ETL Manager
from lichens import EtlManager
em = EtlManager(constr=CONNECTION_STRING, name=ETL_NAME)

# Use Pandera to validate data
from lichens import DataFrameSchema, check_io
schema = DataFrameSchema(...)
validated_df = schema(df)

## Load df to datebase
em.load_df(
        df=df,
        tablename="sample_table",
        schema="public",
        if_exists="raise_error",
        chunksize=500,
        unique_key=["column1", "column2"],
    )

# Update log and archive file
em.update_status(
        filename=f, 
        status=_status, 
        last_log=_last_log)
em.move(
    src=os.path.join(em.src_folder, f),
    status=_status)

# Use state highlighted logger
from lichens.logging import logger as log

# Run as scheduled
## Define a function to be executed
def your_function():
    # Your function logic here
    pass

# Run the function every minute for a total of 5 times
em.run_as_schtask(your_function, '*/1 * * * *', times_=5)
## Decorate the function with the scheduled decorator
@em.scheduled(crontab='*/1 * * * *', times_=5)
def your_scheduled_function():
    # Your function logic here
    pass