Skip to main content

ETL programming in Python

Project description

pygrametl (pronounced py-gram-e-t-l) is a Python framework that provides commonly used functionality for the development of Extract-Transform-Load (ETL) processes. It is open-source and released under a 2-clause BSD license.

When using pygrametl, the developer codes the ETL process in Python code. This turns out to be very efficient, also when compared to drawing the process in a graphical user interface (GUI) like Pentaho Data Integration. It supports both CPython and Jython so existing Java code and JDBC drivers can be used in the ETL program.

Concretely, the developer creates an object for each dimension and fact table. (S)he can then easily add new members by dimension.insert(row) where row is a dict holding the values to insert. This is a very simple example, but pygrametl also supports much more complicated scenarios. For example, it is possible to create a single object for a snowflaked dimension. It is then still possible to add a new dimension member with a single method call as in snowflake.insert(row). This will automatically do the necessary lookups and insertions in the tables participating in the snowflake.

pygrametl also supports slowly changing dimensions. Again, the programmer only has to invoke a single method: scdim.scdensure(row). This will perform the needed updates of both type 1 (i.e., overwrites) and type 2 (i.e., adding new versions).

pygrametl was first made publicly available in 2009. Since then, we have made multiple improvements and added new features. Version 2.6 was released in December 2018. Today, pygrametl is used in production systems in different sectors such as healthcare, finance, and transport.

Installation

pygrametl can be installed from PyPI with the following command:

$ pip install pygrametl

The current development version of pygrametl is available on GitHub:

$ git clone https://github.com/chrthomsen/pygrametl.git

For more information about installation see the Install Guide.

Documentation

The documentation is available in HTML and as a PDF. There are also installation and beginner guides available.

In addition to the documentation, multiple papers have been published about pygrametl. The papers are listed here and provide a more detailed description of the foundational ideas behind pygrametl but is obviously not keep up to date with changes and improvements implemented in the framework, for such see the documentation. If you use pygrametl in academia, please cite the relevant paper(s).

Community

To keep the development of pygrametl open for external participation, we have public mailing lists and use Github. Feel free to ask questions and provide all kinds of feedback:

  • pygrametl-user - For any questions about how to deploy and utilize pygrametl for ETL.

  • pygrametl-dev - For - questions and discussion about the development of pygrametl.

  • Github - Bugs and patches should be submitted to Github as issues and pull requests.

When asking a question or reporting a possible bug in pygrametl, please first verify that the problem still occurs with the latest version of pygrametl. If the problem persists after updating please include the following information, preferably with detailed version information, when reporting the problem:

  • Operating System.

  • Python Implementation.

  • Relational Database Management System.

  • Python Database Connector.

  • A short description of the problem with a minimal code example that reproduces the problem.

We encourage the use of Github and the mailing lists. For discussions not suitable for a public mailing list, you can, however, send us a private email.

Maintainers

pygrametl is maintained at Aalborg University by Christian Thomsen and Søren Kejser Jensen.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pygrametl-2.7.tar.gz (157.8 kB view details)

Uploaded Source

File details

Details for the file pygrametl-2.7.tar.gz.

File metadata

  • Download URL: pygrametl-2.7.tar.gz
  • Upload date:
  • Size: 157.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.0 CPython/3.9.5

File hashes

Hashes for pygrametl-2.7.tar.gz
Algorithm Hash digest
SHA256 8c7f3afa4e8a591110e72fe17d934c5e04bd465e90da9379b2e39603eb293cbb
MD5 1c276ccfd0b03f4eb438b20449054d03
BLAKE2b-256 3024c6a81bfaf4b73e468b1cb7673d17f77379e0e0ca1b1fafc416370e787de5

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page