urgap 3.3.9

No project description provided

Urgap is a cloud-native framework for file-based data engineering, containing abstraction layers for data and meta data, extensive re-run skipping logic and data versioning. Urgap can be incorporated with any scheduling/pipelining tool making pipeline development independent from business logic and data storage, while offering standardized logging and execution, which makes monitoring and debugging easy.

Urgap gives us the governance constraints required for a decentralized data domain autonomy as Urgap will enforce shared common data IO for storage, a common meta data capturing process in form of an interface thus can be plugged into any existing processes and finally global data lineages.

Learn More

Watch our introduction talk urgap - unified resource governance and data provenance by Christian Fufezan to get a comprehensive overview of urgap’s design and capabilities:

How to Setup

Prerequisites

We recommend using a virtual environment for Python projects. This guide uses uv for dependency management.

Installation

Basic Installation (local file system access only):

uv pip install -e .

With Cloud Storage Support:

uv pip install -e ".[cloud]"

With All Optional Dependencies:

uv pip install -e ".[all]"

Available extras include:

• cloud: Azure and Google Cloud storage backends

• all: All optional dependencies

Running Tests

Install test dependencies:

uv pip install pytest

Run the test suite:

pytest tests

Quickstart: Writing Your First Pipeline

The best way to learn urgap is through a complete example. Check out the end-to-end filter CSV pipeline:

• Location: tests/integrationtests/end2end/test_filter_csv_pipeline.py

• What it demonstrates: Complete pipeline setup, node configuration, and execution

• Requirements: Everything needed to run this example is included in the repository

This example can be run entirely on your local machine without any external dependencies.

To run the example:

pytest tests/integrationtests/end2end/test_filter_csv_pipeline.py

Documentation

Please use sphinx in the docs folder

Note

Currently CI does not include pushing the documentation to readthedocs, therefore please #. checkout the repo #. pip install -e . #. cd docs #. make html #. open docs/build/index.html