Urgap is a cloud-native framework for file-based data engineering, containing abstraction layers for data and meta data, extensive re-run skipping logic and data versioning. Urgap can be incorporated with any scheduling/pipelining tool making pipeline development independent from business logic and data storage, while offering standardized logging and execution, which makes monitoring and debugging easy.
Urgap gives us the governance constraints required for a decentralized data domain autonomy as Urgap will enforce shared common data IO for storage, a common meta data capturing process in form of an interface thus can be plugged into any existing processes and finally global data lineages.
Learn More
Watch our introduction talk urgap - unified resource governance and data provenance by Christian Fufezan to get a comprehensive overview of urgap’s design and capabilities:
How to Setup
Prerequisites
We recommend using a virtual environment for Python projects. This guide uses uv for dependency management.
Installation
Basic Installation (local file system access only):
uv pip install -e .With Cloud Storage Support:
uv pip install -e ".[cloud]"With All Optional Dependencies:
uv pip install -e ".[all]"Available extras include:
cloud: Azure and Google Cloud storage backends
all: All optional dependencies
Running Tests
Install test dependencies:
uv pip install pytestRun the test suite:
pytest testsQuickstart: Writing Your First Pipeline
The best way to learn urgap is through a complete example. Check out the end-to-end filter CSV pipeline:
Location: tests/integrationtests/end2end/test_filter_csv_pipeline.py
What it demonstrates: Complete pipeline setup, node configuration, and execution
Requirements: Everything needed to run this example is included in the repository
This example can be run entirely on your local machine without any external dependencies.
To run the example:
pytest tests/integrationtests/end2end/test_filter_csv_pipeline.pyDocumentation
Please use sphinx in the docs folder
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file urgap-3.3.11.tar.gz.
File metadata
- Download URL: urgap-3.3.11.tar.gz
- Upload date:
- Size: 330.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
ccd440df0992d97a781c74ad0eba4941ddaf87dbc6c004232ff3c88f6f529ae1
|
|
| MD5 |
bceec03e4c95c46fd3ce1ffe466d3042
|
|
| BLAKE2b-256 |
9388720bd036dd1d1acd540f4eca98ce4378e5a78f06feb1f6c4d4a99f28334d
|
File details
Details for the file urgap-3.3.11-py2.py3-none-any.whl.
File metadata
- Download URL: urgap-3.3.11-py2.py3-none-any.whl
- Upload date:
- Size: 398.5 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2262456faf64d5fc65e92f17a7d29f034c339d43bb3bac9a098d68419691091a
|
|
| MD5 |
b43eaab408811c64c80f6a6ab7954f99
|
|
| BLAKE2b-256 |
a1bd4db8a6671d6532c483038232854400952d9919d8dc751072dabd34838d57
|
