Extensions

An extensions functions and objects which provides all plug-ins and objects that use for data processing and transformation.

:round_pushpin: Installation

pip install -U ddeutil-extensions

:rocket: Features

This extensions package provides 3 main components:

• Plug-in the extension tasks that use with the Workflow
• Connection and Dataset interface objects
• Schema models

Connection

The connection for worker able to do anything.

conn_postgres_data:
  type: conn.Postgres
  url: 'postgres//username:${ENV_PASS}@hostname:port/database?echo=True&time_out=10'

from ddeutil.extensions.conn import Conn

conn = Conn.from_loader(name='conn_postgres_data', externals={})
assert conn.ping()

Dataset

The dataset is defined any objects on the connection. This feature was implemented on /extensions because it has a lot of tools that can interact with any data systems in the data tool stacks.

ds_postgres_customer_tbl:
  type: dataset.PostgresTbl
  conn: 'conn_postgres_data'
  features:
    id: serial primary key
    name: varchar( 100 ) not null

from ddeutil.extensions.datasets.pg import PostgresTbl

dataset = PostgresTbl.from_loader(name='ds_postgres_customer_tbl', externals={})
assert dataset.exists()

:speech_balloon: Contribute

I do not think this project will go around the world because it has specific propose, and you can create by your coding without this project dependency for long term solution. So, on this time, you can open the GitHub issue on this project :raised_hands: for fix bug or request new feature if you want it.