Urgap is a cloud-native framework for file-based data engineering, containing abstraction layers for data and meta data, extensive re-run skipping logic and data versioning. Urgap can be incorporated with any scheduling/pipelining tool making pipeline development independent from business logic and data storage, while offering standardized logging and execution, which makes monitoring and debugging easy.
Urgap gives us the governance constraints required for a decentralized data domain autonomy as Urgap will enforce shared common data IO for storage, a common meta data capturing process in form of an interface thus can be plugged into any existing processes and finally global data lineages.
Learn More
Watch our introduction talk urgap - unified resource governance and data provenance by Christian Fufezan to get a comprehensive overview of urgap’s design and capabilities:
How to Setup
Prerequisites
We recommend using a virtual environment for Python projects. This guide uses uv for dependency management.
Installation
Basic Installation (local file system access only):
uv syncWith Cloud Storage Support:
uv sync --extra cloudWith All Runtime Extras:
uv sync --extra allAvailable extras include:
server: FastAPI/Flask/uvicorn + MCP server for uctl run
tabular: pandas/polars/pyarrow and Excel engines for tabular unodes
viz: plotly-based reporting and visualizer unodes
github: GitHub-backed file IO
cloud: Azure and Google Cloud storage backends
databases: PostgreSQL backend
docs: Sphinx documentation build
dev: test tooling (pytest) plus all runtime extras
all: every runtime extra above (not dev or docs)
Running Tests
Install test dependencies:
uv sync --extra devRun the test suite:
uv run pytest testsQuickstart: Writing Your First Pipeline
The best way to learn urgap is through a complete example. Check out the end-to-end filter CSV pipeline:
Location: tests/integrationtests/end2end/test_filter_csv_pipeline.py
What it demonstrates: Complete pipeline setup, node configuration, and execution
Requirements: Everything needed to run this example is included in the repository
This example can be run entirely on your local machine without any external dependencies.
To run the example:
pytest tests/integrationtests/end2end/test_filter_csv_pipeline.pyDocumentation
Please use sphinx in the docs folder
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file urgap-3.4.0.tar.gz.
File metadata
- Download URL: urgap-3.4.0.tar.gz
- Upload date:
- Size: 331.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9921a1a220f979ec2f196ac7c2c70db4cb9b5d2b4b60da9eddf9d83dce97531b
|
|
| MD5 |
a4ebbdcc5639ef92a26fbf7a9281de1c
|
|
| BLAKE2b-256 |
7f18329ba05bfd3ba7d458999a5cb07bf504cf6b31cf1f0b0f9b92731efdc233
|
File details
Details for the file urgap-3.4.0-py3-none-any.whl.
File metadata
- Download URL: urgap-3.4.0-py3-none-any.whl
- Upload date:
- Size: 399.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
b051edc46f9053ad360376184d05554026e1af53064fb229edf8c9321e02d2fd
|
|
| MD5 |
39e9b6b57f02e8e845f7a261101f13c1
|
|
| BLAKE2b-256 |
1c392f8a07f1975a1c8e5fc77caba32b7cd5fe2e13c55de2b91042a12ac4d9ad
|
