ralsei 3.0.0.dev3

Build modular data pipelines running inside the postgres database

Ralsei

Ralsei is a Python framework for building modular data pipelines running inside the postgres database.

It was built with use cases such as web scraping in mind, with the philosophy that all artifacts should be stored in the database: from downloaded html to the parsed results

Features

• Based on the jinja-psycopg library, combining type-safe SQL formatting with jinja's template language
• Declarative with minimal boilerplate
• Resumable pipelines, both at row-level and table-level granularity - no need to re-compute or re-download what has already been processed

Installation

pip install ralsei

Tip: consider using PDM or Poetry for project-based dependency management

Quick Start

First, create a script from the following template:

from ralsei import RalseiCli

def make_pipeline(args):
    return {} #  Declare your pipeline in the format "name": Task(...)

if __name__ == "__main__":
    cli = RalseiCli() # Here you can add custom arguments
    cli.run(make_pipeline)

To see some example pipelines, take a look at the Builtin Tasks section of the documentation

Alternatives

• DBT - jinja + SQL based
  more suitable for processing data that you already have
• Kedro - python based, more suitable for processing data that you already have