application framework for ETL(ELT) processing
Table of Contents
- How to Contribute
- Quick Start
- YAML Configuration
- Default ETL Modules
- How to Implement Additional ETL Modules
What is cliboa
cliboa is an application framework which can implement ETL(ELT) processing. It eases the implementation of ETL(ELT) processing. In this case, ETL(ELT) Processing means the processings like fetch, transform and transfer of data between various databases, storages, and other services.
- Python based framework.
- ETL(ELT) processing is executable by YAML based configuration.
- Additional modules for ETL(ELT) processing can be implemented by only a few steps if default modules not enough.
How to Contribute
Available on macOS and any Linux distributions, like Debian, Ubuntu, CentOS, REL, or etc.
python version 3.5 or later and pipenv are required. In the environemnt which pip can be used, execute as below.
sudo pip3 install pipenv sudo pip3 install cliboa
Configuration of a Simple ETL Processing
After installed cliboa, 'cliboadmin' can be used as an administrator command.
Create an executable environment of cliboa by using cliboadmin.
$ cd /usr/local $ sudo cliboadmin init sample $ cd sample $ sudo cliboadmin create simple-etl
Directory tree which was created aforementioned commands is as below.
sample ├── Pipfile ├── bin │ └── clibomanager.py ├── cliboa │ └── conf ├── common │ ├── __init__.py │ ├── environment.py │ └── scenario ├── conf │ ├── cliboa.ini │ └── logging.conf ├── logs ├── project │ └── simple-etl │ ├── scenario │ └── scenario.yml └── requirements.txt
Install PyPI packages
$ cd sample $ pipenv install --dev
$ cd sample $ sudo pip3 install -r requirements.txt
Write a Scenario of ETL Processing
As a simple etl processing, write scenario.yml in simple-etl as below.
The following example is just download a gzip file from the local sftp server, decompress it, and upload it to the local sftp server.
Set an Environment
To make the above scenario available, set a local machine as a sftp server according to respective environments. Also, put "test.csv.gz" under /usr/local.
Execute a Scenario of ETL Processing
After wrote scenario.yml and set the environment, execute a scenario by as below command.
cd sample pipenv run python3 bin/clibomanager.py simple-etl
cd sample python3 bin/clibomanager.py simple-etl
Default ETL Modules
How to Implement Additional ETL Modules
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.