ternyxmimosa 0.7.0

A minimal modern data stack with working data pipelines in a single Docker container.

mimosa

The ELT part of a modern data stack with practical data pipelines using cloud functionality.
Explore the docs »

Report Bug · Request Feature

Table of Contents

1. About the Project
2. Getting Started
   • Prerequisites
   • API Keys
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. Contact

(back to top)

About the Project

The ELT part of a modern data stack with practical data pipelines and reporting using cloud functionality. This is similar in concept to mimodast using alternative software options and cloud functionality.

Mimosa encompasses the ELT (extract load transform) components necessary to generate the webpage found at gas.aspireto.win, providing detailed reports on natural gas storage volumes within the European Union. This process involves retrieving data from a REST API, transforming it, and storing it in a database tailored for reporting purposes.

The source data is published by Gas Infrastructure Europe and exposed in a REST API.

Beyond gas storage data, Mimosa offers a hands-on experience with essential tools:

• 🚀 dlt for smooth data loading.
• 🔍 dbt for powerful data transformation.
• ☁️ MotherDuck for storing the data in a cloud based DuckDB database.

Further the full tech stack used to create the gas.aspireto.win pages is detailed below.

(back to top)

Getting Started

Prerequisites

Setup a Python development environment.

API Keys

Ensure the following sensitive information is securely stored in environment variables or within a .env file:

• To access the GIE Gas Inventory REST API, an API key is necessary. Quickly obtain your API key by signing up for a free GIE account. Once acquired, expose it using the following environment variable:

  • ENV_GIE_XKEY = "YOUR-API-KEY"

• For MotherDuck, you'll need the service token and the database name. Set up the following environment variables to establish the connection:

  • DESTINATION__MOTHERDUCK__CREDENTIALS = "md:///YOUR-DATABASE-NAME?token=YOUR-SERVICE-TOKEN"
  • Please note that the MotherDuck page utilizes a different format, whereas the above format is specifically required for dlt.

(back to top)

Installation

Execute the following command. Consider using a venv.

pip install ternyxmimosa

Alternatively clone this repository and use poetry install. Or pip install from GitHub.

(back to top)

Usage

Command Line

Not currently supported.

As a Python Package

The following sample obtains the storage data for the last available date and stores it in MotherDuck.

import mimosa.cli as GEI

GEI.main()

Tech Stack

These are the technologies driving the content on gas.aspireto.win:

• Google cloud function for the ELT component:
  • The function is a bare bones wrapper around the mimosa Python package (the current repository). The function is in this repository.
  • It is scheduled to run the ELT twice daily (using Google Scheduler and Pub/Sub message).
  • The result is updated data in MotherDuck.
• Reporting notebook
  • Built using the evidence reporting tool, defined in this GitHub repository.
  • Rebuild and published to a web host using a GitHub workflow.
    • Run on a twice daily schedule. The workflow is defined in the notebook repository.

NOTE: As of November 2023 it is possible to fully deploy this stack without breaking the bank (using free tiers of the cloud services used). Dive into our GitHub repository and the linked ones for the Google Function and Evidence notebook, where all the code awaits. 🚀

Sentry

To enable logging iusing sentry.io specify the environment variable RUNTIME__SENTRY_DSN

(back to top)

NOTE: For some reason the environment variable DESTINATION__MOTHERDUCK__CREDENTIALS is oftentimes incorrectly set between runs when using the dev container. Use unset DESTINATION__MOTHERDUCK__CREDENTIALS to clear the environment variable.

Roadmap

Consider:

• Get source data (Using REST API)
• Transform data, possibly SQL Mesh or dbt.
  • Create data vault transformations (https://automate-dv.readthedocs.io/en/latest/).
• dlt update/error messages using Slack
• Storage (currently local DuckDB, maybe consider some cloud alternative. Though that would stray from the data stack in a Docker concept.) (MotherDuck)
• Scheduling Tool (Google Cloud Scheduler)
• Reporting tool (Metabase?) (Evidence.dev in separate repository)
• Bare bones CLI

(back to top)

Contributing

Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also open a feature request or bug report. Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

Contact

Project Link: mimosa

(back to top)