Skip to main content

Ubunye Engine

Ubunye (oo-BOON-yeh): isiZulu for "unity"

One framework. Every pipeline. Any environment.

Docs • Docs map • Try it • Community

Write a task once, run it unchanged on pandas, Spark, Databricks or any cloud; every run leaves a record, and ubunye prove checks the rows match


A data pipeline moves data from one place to another (a database to a file, a REST API to a data warehouse) and usually changes it on the way. Most of the work is plumbing: connections, credentials, a framework's quirks, the same read, transform, write scaffold again and again.

Ubunye Engine writes that plumbing for you. You describe what to read and write in a short YAML file and put your change to the data in a normal Python class. Ubunye does the connections, the compute engine (Apache Spark, or pandas on a laptop with no Java) and the read and write loop. The same folder runs on your laptop today and on a cluster tomorrow, with no code changes.

What you get:

  • One standard layout. Every project is use case, pipeline, task, run by one CLI. The words are explained in Concepts.
  • A record of every run. Each run can leave a run record with a hash of every row it read and wrote, so two runs can be compared exactly.
  • Checks before you run. ubunye plan and ubunye doctor say what would fail, and why, before any data moves. Errors say what went wrong and suggest the fix.
  • Open plugins. Connectors, backends, model registries and run record stores are added from outside, with no engine edits.
  • Types that ship. The package carries its type information, and a type checker guards every merge.

Try it

On a laptop, with no Java and no cloud account:

pip install "ubunye-engine[pandas]"
ubunye init -d pipelines -u demo -p starter -t filter_adults
ubunye plan -d pipelines -u demo -p starter -t filter_adults --backend pandas
ubunye run -d pipelines -u demo -p starter -t filter_adults --backend pandas --lineage
ubunye lineage list -d pipelines -u demo -p starter -t filter_adults

init makes a task folder with a sample CSV, plan checks it without moving data, run writes Parquet and keeps a run record, and lineage list shows that record. The test suite runs these four commands exactly as written. What each one did, and how to run the same folder on Spark: the Quickstart.


Docs

The full docs live at ubunye-ai-ecosystems.github.io/ubunye_engine. These are the pages most people need:

You want to Read
Install it Installation
Run a first task Quickstart
Learn the words (use case, pipeline, task, profile, run record) Concepts
Write a config.yaml Config overview, then Inputs and outputs
Check the data a task writes Expectations
Read what a run did (the run record) Lineage commands and the run record
Run on pandas or Spark Execution backends
Look up a command CLI reference
Call it from Python or a notebook Python API
Add a language model step Language model steps
Copy a working example Examples
Read from or write to a system Connectors
Deploy it Deployment
Fix something that went wrong Troubleshooting, then Errors
Move from 0.8 to 0.9 Upgrading
See what changed Changelog

Worked examples, grouped by where they run (Databricks, a laptop, Docker, Kubernetes), live in ubunye-examples.


Proof

One task, unchanged, gave the same rows on pandas (no Java), local Spark, Kubernetes, AWS Glue, GCP Dataproc, Azure Container Apps and Databricks: the same data hash in all seven, compared from their run records by ubunye prove. See the latest results.


What Ubunye is not

  • Not an agent framework: use LangChain or CrewAI for that.
  • Not an orchestrator: use Airflow, Prefect or Dagster (ubunye export writes an Airflow DAG or a Databricks job for you).
  • Not a compute engine: it runs on Spark, or on pandas for laptop-sized data.

Ubunye is the standard layer between your data sources and your applications. It makes the plumbing boring so you can focus on what matters.

Not yet: data drift detection is a pattern you write in a task, not a built-in feature. EMR Serverless is built but has not run the proving workload, and Hadoop/YARN is not tested. Neither is claimed until a run says so.


Get involved

New connectors, bug fixes, typos, or just telling us what you are building: all help.

License

MIT License


Built with 🇿🇦 by Ubunye AI Ecosystems

Metadata

Release files for ubunye-engine 0.9.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ubunye-engine 0.9.1
File Size Uploaded Provenance Actions
ubunye_engine-0.9.1.tar.gz 427.1 kB With provenance Details for ubunye_engine-0.9.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ubunye-engine 0.9.1
File Interpreter ABI Platform Actions
ubunye_engine-0.9.1-py3-none-any.whl Python 3 none any Details for ubunye_engine-0.9.1-py3-none-any.whl

Total release size: 932.9 kB

Release files / ubunye_engine-0.9.1.tar.gz

Download URL ubunye_engine-0.9.1.tar.gz
Size 427.1 kB
Tags Source
SHA-256 checksum
How to use checksums
005b267837671c80d4c95bdfa7ff6a2d6fe79b492d682de354ff614781fe31d1
BLAKE2b-256 checksum
How to use checksums
c03dadaed4cc9750f5215b81c05dc30cd79e8a5840bef8663064a4c4aac4115b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.

Transparency log

Release files / ubunye_engine-0.9.1-py3-none-any.whl

Download URL ubunye_engine-0.9.1-py3-none-any.whl
Size 505.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1a096418fc179c3244b617ae6f2d5da406f217a11245508b0e28644d02794707
BLAKE2b-256 checksum
How to use checksums
a743d6857716e43b78cb4082c2bdd4282328626dfdb33fc82d2a8938bc844014
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.

Transparency log

Security for ubunye-engine 0.9.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for ubunye_engine-0.9.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page