Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Pyelt is a DDL and ETL framework for creating and filling data vault datawarehouses on a postgress database.

Project Description

Usage

This example will create and fill the historical staging area:

pipeline = Pipeline(config)
pipe = pipeline.get_or_create_pipe('test_source', source_config)

source_file = CsvFile(get_root_path() + '/sample_data/patienten1.csv', delimiter=';')
source_file.reflect()
source_file.set_primary_key(['patientnummer'])
mapping = SourceToSorMapping(source_file, 'persoon_hstage', auto_map=True)
pipe.mappings.append(mapping)

pipeline.run()

More examples can be found on the GitHub repository of NL Healthcare.

Introduction

Pyelt is a Python DDL and ETL framework for creating and loading Data Vaults for datawarehousing.

Pyelt supports several data-layers, including Source-of-Record (SOR), Raw datavault (RDV), Business datavault (BDV) and Datamarts (DM)

Pyelt can import data from several different source systems such as fixed length files, csv-files, and different databases.

Pyelt is developed to run on a postgreSQL database.

Pyelt uses the SQLAlchemy.core only for the connection and for reflection. All other SQL statements (ddl, copy, insert and update statements) are created by the pyelt framework itself.

Write your own mappings to transfer and transform data from sources via staging into the data ware house.

Background

The pyelt framework is presently under development at NL Healthcare, with the aim to implement our next-generation datawarehouse (DWH2.0). It serves as the foundation for our work in the area of clinical business intelligence (CBI) and machine-learning.

Architectural cornerstones of this project are:

Release History

Release History

This version
History Node

0.9.4.6a0

History Node

0.9.4.5a0

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
pyelt-0.9.4.6a0.zip (71.2 kB) Copy SHA256 Checksum SHA256 Source Sep 16, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting