Skip to main content

Ubunye Engine

Ubunye (oo-BOON-yeh): isiZulu for "unity"

One framework. Every pipeline. Any environment.

Docs • Docs map • Try it • Community

Write a task once, run it unchanged on pandas, Spark, Databricks or any cloud; every run leaves a record, and ubunye prove checks the rows match


A data pipeline moves data from one place to another (a database to a file, a REST API to a data warehouse) and usually changes it on the way. Most of the work is plumbing: connections, credentials, a framework's quirks, the same read, transform, write scaffold again and again.

Ubunye Engine writes that plumbing for you. You describe what to read and write in a short YAML file and put your change to the data in a normal Python class. Ubunye does the connections, the compute engine (Apache Spark, or pandas on a laptop with no Java) and the read and write loop. The same folder runs on your laptop today and on a cluster tomorrow, with no code changes.

What you get:

  • One standard layout. Every project is use case, pipeline, task, run by one CLI. The words are explained in Concepts.
  • A record of every run. Each run can leave a run record with a hash of every row it read and wrote, so two runs can be compared exactly.
  • Checks before you run. ubunye plan and ubunye doctor say what would fail, and why, before any data moves. Errors say what went wrong and suggest the fix.
  • Open plugins. Connectors, backends, model registries and run record stores are added from outside, with no engine edits.
  • Types that ship. The package carries its type information, and a type checker guards every merge.

Try it

On a laptop, with no Java and no cloud account:

pip install "ubunye-engine[pandas]"
ubunye init -d pipelines -u demo -p starter -t filter_adults
ubunye plan -d pipelines -u demo -p starter -t filter_adults --backend pandas
ubunye run -d pipelines -u demo -p starter -t filter_adults --backend pandas --lineage
ubunye lineage list -d pipelines -u demo -p starter -t filter_adults

init makes a task folder with a sample CSV, plan checks it without moving data, run writes Parquet and keeps a run record, and lineage list shows that record. The test suite runs these four commands exactly as written. What each one did, and how to run the same folder on Spark: the Quickstart.


Docs

The full docs live at ubunye-ai-ecosystems.github.io/ubunye_engine. These are the pages most people need:

You want to Read
Install it Installation
Run a first task Quickstart
Learn the words (use case, pipeline, task, profile, run record) Concepts
Write a config.yaml Config overview, then Inputs and outputs
Check the data a task writes Expectations
Read what a run did (the run record) Lineage commands and the run record
Run on pandas or Spark Execution backends
Look up a command CLI reference
Call it from Python or a notebook Python API
Add a language model step Language model steps
Copy a working example Examples
Read from or write to a system Connectors
Deploy it Deployment
Fix something that went wrong Troubleshooting, then Errors
Move from 0.8 to 0.9 Upgrading
See what changed Changelog

Worked examples, grouped by where they run (Databricks, a laptop, Docker, Kubernetes), live in ubunye-examples.


Proof

One task, unchanged, gave the same rows on pandas (no Java), local Spark, Kubernetes, AWS Glue, GCP Dataproc, Azure Container Apps and Databricks: the same data hash in all seven, compared from their run records by ubunye prove. See the latest results.


What Ubunye is not

  • Not an agent framework: use LangChain or CrewAI for that.
  • Not an orchestrator: use Airflow, Prefect or Dagster (ubunye export writes an Airflow DAG or a Databricks job for you).
  • Not a compute engine: it runs on Spark, or on pandas for laptop-sized data.

Ubunye is the standard layer between your data sources and your applications. It makes the plumbing boring so you can focus on what matters.

Not yet: data drift detection is a pattern you write in a task, not a built-in feature. EMR Serverless is built but has not run the proving workload, and Hadoop/YARN is not tested. Neither is claimed until a run says so.


Get involved

New connectors, bug fixes, typos, or just telling us what you are building: all help.

License

MIT License


Built with 🇿🇦 by Ubunye AI Ecosystems

Metadata

Release files for ubunye-engine 0.9.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ubunye-engine 0.9.3
File Size Uploaded Provenance Actions
ubunye_engine-0.9.3.tar.gz 445.1 kB With provenance Details for ubunye_engine-0.9.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ubunye-engine 0.9.3
File Interpreter ABI Platform Actions
ubunye_engine-0.9.3-py3-none-any.whl Python 3 none any Details for ubunye_engine-0.9.3-py3-none-any.whl

Total release size: 971.7 kB

Release files / ubunye_engine-0.9.3.tar.gz

Download URL ubunye_engine-0.9.3.tar.gz
Size 445.1 kB
Tags Source
SHA-256 checksum
How to use checksums
b1076c28be6c1dc8ac1ca97bef8dba79544d04fd616a4bee2452913905d6b3c7
BLAKE2b-256 checksum
How to use checksums
62ef806d7c3100412d4eab8cadff9c37c994ff03d136e1b2cd3eba7c85919f45
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.

Transparency log

Release files / ubunye_engine-0.9.3-py3-none-any.whl

Download URL ubunye_engine-0.9.3-py3-none-any.whl
Size 526.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
49f2c5eeff38533c9ca742f4bd005ec86c0a9be97af3b9fc8de89b7c1a8bdf18
BLAKE2b-256 checksum
How to use checksums
9c15c36bf462580832db8d16a7f7cd2ab56c9b388b5f9507d80c2519d5477ac9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.

Transparency log

Security for ubunye-engine 0.9.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for ubunye_engine-0.9.3.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 10, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page