Skip to main content

application framework for ETL(ELT) processing

Project description

PyPI PyPI - Implementation PyPI - Python Version GitHub Actions Code Style: black Contributions Welcome Repo Size Gitter

Table of Contents

Introduction

What is cliboa

cliboa is an application framework which can implement ETL(ELT) processing. It eases the implementation of ETL(ELT) processing. In this case, ETL(ELT) Processing means the processings like fetch, transform and transfer of data between various databases, storages, and other services.

Features

  • Python based framework.
  • ETL(ELT) processing is executable by YAML based configuration.
  • Additional modules for ETL(ELT) processing can be implemented by only a few steps if default modules not enough.

Manual

See MANUAL.md

How to Contribute

See CONTRIBUTING.md

Quick Start

Requirements

Available on macOS and any Linux distributions, like Debian, Ubuntu, CentOS, REL, or etc.

Install cliboa

python version 3.5 or later and pipenv are required. In the environemnt which pip can be used, execute as below.

sudo pip3 install pipenv
sudo pip3 install cliboa

Configuration of a Simple ETL Processing

After installed cliboa, 'cliboadmin' can be used as an administrator command.

Create an executable environment of cliboa by using cliboadmin.

$ cd /usr/local
$ sudo cliboadmin init sample
$ cd sample
$ sudo cliboadmin create simple-etl

Directory Tree

Directory tree which was created aforementioned commands is as below.

sample
├── Pipfile
├── bin
│   └── clibomanager.py
├── cliboa
│   └── conf
├── common
│   ├── __init__.py
│   ├── environment.py
│   └── scenario
├── conf
│   ├── cliboa.ini
│   └── logging.conf
├── logs
├── project
│   └── simple-etl
│       ├── scenario
│       └── scenario.yml
└── requirements.txt

Install PyPI packages

$ cd sample
$ pipenv install --dev

or

$ cd sample
$ sudo pip3 install -r requirements.txt

Write a Scenario of ETL Processing

As a simple etl processing, write scenario.yml in simple-etl as below.

The following example is just download a gzip file from the local sftp server, decompress it, and upload it to the local sftp server.

See Examples

Set an Environment

To make the above scenario available, set a local machine as a sftp server according to respective environments. Also, put "test.csv.gz" under /usr/local.

Execute a Scenario of ETL Processing

After wrote scenario.yml and set the environment, execute a scenario by as below command.

cd sample
pipenv run python3 bin/clibomanager.py simple-etl

or

cd sample
python3 bin/clibomanager.py simple-etl

YAML Configuration

see yaml_configuration.md

Default ETL Modules

see default_etl_modules.md

How to Implement Additional ETL Modules

see additional_etl_modules.md

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cliboa-2.1.0b0.tar.gz (55.6 kB view hashes)

Uploaded source

Built Distribution

cliboa-2.1.0b0-py3-none-any.whl (106.2 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page