deflow 0.0.6

Lightweight Declarative Data Framework

DeFlow

A Lightweight Declarative Data Framework that allow you to run data pipelines by YAML config template.

[!NOTE] I want to use this project is the real-world use-case for my Workflow package that able to handle production data pipeline with the DataOps strategy.

[!WARNING] This framework does not allow you to custom your pipeline yet. If you want to create your workflow, you can implement it by your custom template reference this package.

In my opinion, I think it should not create duplicate workflow codes if I can write with dynamic input parameters on the one template workflow that just change the input parameters per use-case instead. This way I can handle a lot of logical workflows in our orgs with only metadata configuration. It called Metadata Driven Data Workflow.

📦 Installation

pip install -U deflow

Support data framework version:

Version	Supported	Description
1	Progress	Large scale base on stream, group, process, and routing.
2	Progress	Medium scale base on pipeline, and node.
3	Progress	Lightweight base on dag, and task.

[!NOTE] I think it should stop with 3 versions of data framework.

:dart: Framework

⭕ Version 1

[!NOTE] This project will create the data framework Version 1 first.

After initialize your data framework project with Version 1, your data pipeline config files will store with this file structure:

conf/
 ├─ routes/
 │   ╰─ routing.yml
 ├─ shared/
 │   ├─ { c_conn_01 }.yml
 │   ╰─ { c_conn_02 }.yml
 ├─ stream/
 │   ╰─ { s_stream_01 }/
 │       ├─ { g_group_01 }.tier.priority/
 │       │   ├─ { p_proces_01 }.yml
 │       │   ╰─ { p_proces_02 }.yml
 │       ├─ { g_group_02 }.tier.priority/
 │       │   ├─ { p_proces_01 }.yml
 │       │   ╰─ { p_proces_02 }.yml
 │       ╰─ config.yml
 ╰─ .confignore

⭕ Version 2

After initialize your data framework project with Version 2, your data pipeline config files will store with this file structure:

conf/
 ├─ pipeline/
 │   ╰─ { p_pipe_01 }/
 │       ├─ config.yml
 │       ├─ { n_node_01 }.yml
 │       ╰─ { n_node_02 }.yml
 ╰─ .confignore

⭕ Version 3

[!NOTE] This version is the same DAG and Task strategy like Airflow.

conf/
 ├─ dag/
 │   ╰─ { dag_cm_d }/
 │       ├─ assets/
 │       │   ├─ { some-asset }.sql
 │       │   ╰─ { some-asset }.json
 │       ├─ config.yml
 │       ╰─ variables.yml
 ╰─ .confignore

Getting Started

You can run the data flow by:

from deflow.flow import Flow
from ddeutil.workflow import Result

flow: Result = (
    Flow(name="s_stream_01", version="v1")
    .option("conf_paths", ["./data/conf"])
    .run(mode="N")
)

:cookie: Configuration

This package configuration:

Name	Component	Default	Description
DEFLOW_CORE_CONF_PATH	CORE	./conf	A config path to get data framework configuration.
DEFLOW_CORE_VERSION	CORE	v1	A specific data framework version.
DEFLOW_CORE_REGISTRY_CALLER	CORE	.	A registry of caller function.

Relate workflow configuration that will impact this package:

💬 Contribute

I do not think this project will go around the world because it has specific propose, and you can create by your coding without this project dependency for long term solution. So, on this time, you can open the GitHub issue on this project 🙌 for fix bug or request new feature if you want it.