Urgap is a cloud-native framework for file-based data engineering, containing abstraction layers for data and meta data, extensive re-run skipping logic and data versioning. Urgap can be incorporated with any scheduling/pipelining tool making pipeline development independent from business logic and data storage, while offering standardized logging and execution, which makes monitoring and debugging easy.
Urgap gives us the governance constraints required for a decentralized data domain autonomy as Urgap will enforce shared common data IO for storage, a common meta data capturing process in form of an interface thus can be plugged into any existing processes and finally global data lineages.
Learn More
Watch our introduction talk urgap - unified resource governance and data provenance by Christian Fufezan to get a comprehensive overview of urgap’s design and capabilities:
How to Setup
Prerequisites
We recommend using a virtual environment for Python projects. This guide uses uv for dependency management.
Installation
Basic Installation (local file system access only):
uv syncWith Cloud Storage Support:
uv sync --extra cloudWith All Runtime Extras:
uv sync --extra allAvailable extras include:
server: FastAPI/Flask/uvicorn + MCP server for uctl run
tabular: pandas/polars/pyarrow and Excel engines for tabular unodes
viz: plotly-based reporting and visualizer unodes
github: GitHub-backed file IO
cloud: Azure and Google Cloud storage backends
databases: PostgreSQL backend
docs: Sphinx documentation build
dev: test tooling (pytest) plus all runtime extras
all: every runtime extra above (not dev or docs)
Running Tests
Install test dependencies:
uv sync --extra devRun the test suite:
uv run pytest testsQuickstart: Writing Your First Pipeline
The best way to learn urgap is through a complete example. Check out the end-to-end filter CSV pipeline:
Location: tests/integrationtests/end2end/test_filter_csv_pipeline.py
What it demonstrates: Complete pipeline setup, node configuration, and execution
Requirements: Everything needed to run this example is included in the repository
This example can be run entirely on your local machine without any external dependencies.
To run the example:
pytest tests/integrationtests/end2end/test_filter_csv_pipeline.pyDocumentation
Please use sphinx in the docs folder
Release files for urgap 3.4.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| urgap-3.4.1.tar.gz | 331.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| urgap-3.4.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 730.7 kB
Release files / urgap-3.4.1.tar.gz
| Download URL | urgap-3.4.1.tar.gz |
|---|---|
| Size | 331.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
59610d74d72d42557a352dadb6c45c61b6ea342ca4a3dcabfff4be9d8d479fc2
|
|
BLAKE2b-256 checksum How to use checksums |
44357702adce29890cf5f7363755fef72d8b15479e10dd791935608512edb50d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / urgap-3.4.1-py3-none-any.whl
| Download URL | urgap-3.4.1-py3-none-any.whl |
|---|---|
| Size | 399.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6fe1290da6b013692aef370cbe5dcb62ed54193746b66791c9711662833eea9d
|
|
BLAKE2b-256 checksum How to use checksums |
729ddf660f90b5b9d877ee5cce5ace0e249aa8624889e87795c0d2a4610e1c5c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
