Skip to main content

Data needs a Makefile.

Project description

Don't ETL or ELT. LET your data be free.

VulknData Eruptr

Data needs a Makefile.

https://tonsky.me/blog/disenchantment/

Eruptr is an extensible, model and configuration driven data management system targeting ETL, DataOps and Analytics Engineering.

The initial target for Eruptr is the ClickHouse OLAP engine however it is being adapted and extended to address other modern data eco-systems.

Why

The main data processing mechanism is over untyped data using existing decades old Unix/Linux tools. One of the key goals of Eruptr in this regard is to force us to ask honest questions about existing data processing mechanisms. Do we need a heavy, Java-based system to process our data or does awk solve the problem for 95% of our data in 10% of the time? What about heavily typed data processing systems in Python that pass dictionaries or tuples around with mountains of code on Kubernetes? Is that necessary? Or could it be dealt with via grep on a micro instance?

Initial Release

The initial release is an alpha release. Tests have not been established and there are quirks/issues in some components or functionality that differs to the documentation. These issues will be rectified prior to Christmas 2020 with a proper production release in January 2021.

Roadmap

  • Generator modules with daemon mode
  • Scheduler/task controller with multi-processing schedulers
  • Engine support for PostgreSQL, MySQL, RedShift and Google BigQuery
  • Additional task, pipe and io modules
  • Analytics / Data Engineering projects
  • Database reflection with automatic model/schema migrations
  • A web UI for managing workflows
  • Analytics engineering workflows including documentation
  • Graph scheduling with multiple flows
  • Marrying eco-systems - Vulkn and the upcoming Vulkn Server

Documentation

Documentation is available at http://docs.vulkndata.io/eruptr/.

Installation

Eruptr has only been tested with recent versions of Ubuntu and Python 3.7. You will need to have a working Python 3.7.x environment with pip.

Ensure you have installed ClickHouse the clickhouse-client program including clickhouse-local:

sudo apt install clickhouse-client clickhouse-common

Installation with pip

  1. Install Eruptr via pip.
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install -y python3.7 python3.7-dev python3-pip
sudo python3.7 -m pip install eruptr

Installation from source (for developers)

  1. Install Eruptr via git clone.
git clone https://github.com/VulknData/eruptr.git
cd eruptr
  1. Install required packages. Note that it may make sense to do this within a virtual environment.
pip install -r requirements.txt
  1. You can start using Eruptr via the eruptr script:
cd scripts
source env.sh
./eruptr --help

Getting Started

Let's use one of the provided examples. This assumes you have a ClickHouse server running on your local system listening on standard ports with no default credentials (default/empty password).

simple1.yaml:

name: Simple1 Batch Job
shard: clickhouse://localhost/test
workflow:
  - drop:
      executor: StepExecutor
  - create:
      executor: StepExecutor
      enabled: true
  - data:
      executor: StepExecutor
  - pre:
      executor: StepExecutor
      enabled: true
  - input:
      executor: UnixPipeExecutor
      enabled: true
  - post:
      executor: StepExecutor
  - clean:
      executor: StepExecutor
data:
  - tasks.file.write:
      path: system_metrics.csv
      data: |
        "device1","1970-04-27 03:46:40",32.3,"temperature"
        "device1","1970-04-27 03:46:40",10.2,"watts"
  - tasks.pack.gz:
      input_file: system_metrics.csv
      output_file: system_metrics.csv.gz
pre:
  - TRUNCATE TABLE simple1.system_metrics
input:
  - io.file.read: system_metrics.csv.gz
  - pipes.unpack.gz
  - io.clickhouse.write:
      table: simple1.system_metrics
      format: formats.clickhouse.CSV
post:
  - tasks.file.delete: system_metrics.csv.gz
create:
  - CREATE DATABASE IF NOT EXISTS simple1
  - |
    CREATE TABLE IF NOT EXISTS simple1.system_metrics
    (
      device String,
      epoch_dt DateTime,
      value Float32,
      tag String
    ) ENGINE = Log
clean:
  - DROP TABLE IF EXISTS simple1.system_metrics
  - DROP DATABASE IF EXISTS simple1

First lets generate some test data:

eruptr load --conf ../examples/simple1/simple1.yaml --log-level INFO --flows data

VulknData Eruptr (C) 2020 VulknData, Jason Godden

GPLv3 - see https://github.com/VulknData/eruptr/COPYING

12/01/2020 09:14:40 PM - INFO - Rendering configuration
12/01/2020 09:14:40 PM - INFO - Running "Simple1 Batch Job"
12/01/2020 09:14:40 PM - INFO - Executing data section
12/01/2020 09:14:40 PM - INFO - StepExecutor: tasks.file.write({'path': 'system_metrics.csv', 'data': '"device1","1970-04-27 03:46:40",32.3,"temperature"\n"device1","1970-04-27 03:46:40",10.2,"watts"\n'}) -> tasks.pack.gz({'input_file': 'system_metrics.csv', 'output_file': 'system_metrics.csv.gz'})
12/01/2020 09:14:40 PM - INFO - Successfully completed "Simple1 Batch Job"

OK - Simple1 Batch Job - SUCCESS

Using the --flows option we've told eruptr to execute the data flow only.

Great. This has provided us with a simple dataset we can import. If we don't specify any flows eruptr automatically runs only the enabled workflows in the order they're defined (create, pre and input).

eruptr load --conf ../examples/simple1/simple1.yaml --log-level INFO

VulknData Eruptr (C) 2020 VulknData, Jason Godden

GPLv3 - see https://github.com/VulknData/eruptr/COPYING

12/01/2020 09:24:04 PM - INFO - Rendering configuration
12/01/2020 09:24:04 PM - INFO - Running "Simple1 Batch Job"
12/01/2020 09:24:04 PM - INFO - Executing create section
12/01/2020 09:24:04 PM - INFO - StepExecutor: tasks.clickhouse.execute(CREATE DATABASE IF NOT EXISTS simple1) -> tasks.clickhouse.execute(CREATE TABLE IF NOT EXISTS simple1.system_metrics ( device String, epoch_dt DateTime, value Float32, tag String ) ENGINE = Log )
12/01/2020 09:24:04 PM - INFO - Executing pre section
12/01/2020 09:24:04 PM - INFO - StepExecutor: tasks.clickhouse.execute(TRUNCATE TABLE simple1.system_metrics)
12/01/2020 09:24:04 PM - INFO - Executing input section
12/01/2020 09:24:04 PM - INFO - UnixPipeExecutor: <function read at 0x7f9e0d61ed90>(run='system_metrics.csv.gz', connection='clickhouse://localhost/test' | <function <lambda> at 0x7f9e0d62bd08>(run='None', connection='clickhouse://localhost/test' | <function write at 0x7f9e0d61ed08>(connection='clickhouse://localhost/test', table='simple1.system_metrics', format='formats.clickhouse.CSV'
12/01/2020 09:24:04 PM - INFO - Successfully completed "Simple1 Batch Job"

OK - Simple1 Batch Job - SUCCESS

And what can we see in ClickHouse?

hulk :) select * from simple1.system_metrics format CSV;

SELECT *
FROM simple1.system_metrics
FORMAT CSV

Query id: f4c471b1-f32a-4636-8e62-d923e5dc10c4

"device1","1970-04-27 03:46:40",32.3,"temperature"
"device1","1970-04-27 03:46:40",10.2,"watts"

2 rows in set. Elapsed: 0.005 sec. 

Perfect. So eruptr has created the necessary database, then table(s) and then extracted and loaded our data.

This is a trivial example though. Explore the documentation further to see how you can create complex workflows with everything from custom shell commands to embedded Python all driven by simple YAML. Or combine Mako and Jinja to create re-usable macros for processing your data.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

eruptr-20.12.2.tar.gz (29.8 kB view details)

Uploaded Source

Built Distribution

eruptr-20.12.2-py3-none-any.whl (64.3 kB view details)

Uploaded Python 3

File details

Details for the file eruptr-20.12.2.tar.gz.

File metadata

  • Download URL: eruptr-20.12.2.tar.gz
  • Upload date:
  • Size: 29.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/50.3.2 requests-toolbelt/0.9.1 tqdm/4.54.0 CPython/3.7.3

File hashes

Hashes for eruptr-20.12.2.tar.gz
Algorithm Hash digest
SHA256 6f07fe5f280596950eae9fd03282071eacecce74a9812ae79f10c49a8ec9fe82
MD5 d0d8e38ff94c1e8681951b5e70daab4b
BLAKE2b-256 a644c5c1c0514c9f4b7337f2f2d9641d3f58fb68114911d8fdd7d4497906de27

See more details on using hashes here.

File details

Details for the file eruptr-20.12.2-py3-none-any.whl.

File metadata

  • Download URL: eruptr-20.12.2-py3-none-any.whl
  • Upload date:
  • Size: 64.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.6.1 requests/2.25.0 setuptools/50.3.2 requests-toolbelt/0.9.1 tqdm/4.54.0 CPython/3.7.3

File hashes

Hashes for eruptr-20.12.2-py3-none-any.whl
Algorithm Hash digest
SHA256 68d60a585b29df0ababe2152079143f9f795f3cc458a2ddd6e34607c8d4bbc70
MD5 ba59778bcfa27b1d820b1755e14484da
BLAKE2b-256 7ef03d6f4484a0ede11be7be733f27f137e82315a89079f6230b43ee8bd31bc0

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page