flowform 0.0.1

Flowform is a declarative configuration language for your data pipelines

Flowform

Flowform is a declarative configuration language for your data pipelines.

For example: you can specify that you want to load data from a PostgreSQL table, perform transformation and load the result to a csv file using this config:

extract:
  connector: postgres

transform:
  getBestSongs:
    getAllSongs: select * from artist
    getBest: where rating > 8

load:
  getBestSongs:
    format: csv
    fileName: result

How does it work?

block	explanation
connector: postgres	specifyies the data source
getBestSongs	task name, you can name it anything
getAllSongs	subquery name, you can name it anything
load:getBestSongs	this loads the result from getBestSongs task

NOTE: Subqueries get appended with the previous subquery.

Configuration

profiles/profiles.yaml specifies the data source configs.

demo/demo.yaml specifies the yaml file for your pipeline.

Installation

pip install flowform

How to run

ff run --file=<script>.yaml

Running demo

You will require postgres installed on your system. In psql you can run following command to load demo table

\i <Path to this directory>/demo/demo.sql

or manually run sql commands in demo.sql to build example table.

Then you need to run the command:

 ff run --file=./demo/demo.yaml 

Support Status

Currently, supports extracting from PostgresSQL and exporting to a file.