ubunye-engine 0.7.0

Ubunye (oo-BOON-yeh) — isiZulu for "unity"

One framework. Every pipeline. Any environment.

Docs • Quickstart • Why Ubunye • Community

Hey there 👋

A data pipeline is a program that moves data from one place to another — a database to a file, a REST API to a data warehouse — and usually reshapes the data along the way. Building one from scratch is mostly plumbing: wire up the connection, juggle credentials, learn a framework's quirks, write the same "read → transform → write" scaffold for the tenth time this year. It's a lot of glue code standing between you and the three lines that actually matter.

Ubunye Engine writes that plumbing for you. You describe the pipeline in a short YAML file and put your transformation in a normal Python class. Ubunye takes care of connections, the compute engine (Apache Spark), and the read/write loop.

Same pipeline runs on your laptop today and on a production cluster tomorrow, with no code changes. That sentence is tested, not hoped: a build job runs one pipeline on six environments and fails if the outputs differ by a single byte.

What the engine gives you today

• Proven portability. The same task runs on a laptop, in Docker, on Kubernetes, against object storage, through the cloud submit path, and on Databricks. One output hash across all of them, checked on every change.
• Models saved anywhere. The model registry writes to a local folder, a Databricks volume, S3 or GCS, chosen purely by the path. New storage kinds are one class and one entry point.
• A truly open plugin system. Connectors, storage backends and registries are all added from the outside, with no engine edits and no inheritance required. A test proves it with a connector the engine has never seen.
• Types that ship. The package carries its type information, and a type checker guards every merge.
• Errors that help. Failures say what went wrong, show the context, and suggest the fix.
• Nothing here is decorative. Eleven worked examples have been run for real. Claims that could not be executed were removed from these docs rather than left to mislead.

Quickstart

On a laptop, with no Java and no cloud account:

pip install "ubunye-engine[pandas]"

ubunye init -d pipelines -u demo -p starter -t filter_adults
ubunye plan -d pipelines -u demo -p starter -t filter_adults --backend pandas
ubunye run -d pipelines -u demo -p starter -t filter_adults --backend pandas --lineage
ubunye lineage list -d pipelines -u demo -p starter -t filter_adults

These four commands are run by the test suite exactly as written. init makes a folder, and the folder is the whole task:

pipelines/demo/starter/filter_adults/
  config.yaml            what to read and write
  transformations.py     your code
  data/people.csv        a small sample to start from

plan checks it without moving any data, run reads the CSV and writes Parquet, and lineage list shows the record the run left, including a hash of every row written. The code is one line:

class FilterAdults(Task):
    def transform(self, sources):
        people = sources["people"]
        return {"adults": people[people["age"] >= 18]}

That line means the same thing in pandas and in Spark. With Java installed (pip install "ubunye-engine[spark]"), leave out --backend pandas and the same folder runs on Spark, leaving the same data hash. On Databricks, call it from a notebook and the notebook's session is used:

import ubunye
outputs = ubunye.run_task("pipelines/demo/starter/filter_adults")

Step by step: the Quickstart. Realistic end to end examples live in ubunye-examples.

Why Ubunye

We've all been there. You join a new team, open the repo, and find five Spark projects — each structured differently, each with its own way of handling configs, credentials, and deployment. One uses a JSON file, another has everything hardcoded, a third has a 300-line bash script that "Dave wrote and it just works."

Ubunye says: let's agree on how pipelines look. One folder structure. One config format. One CLI. Whether you're building an ETL job, a feature pipeline, or an ML training run.

How It Works

Three simple ideas:

Config over code. Your pipeline is a YAML file. Inputs, outputs, Spark settings, scheduling — all declared, not coded.

Plugins for everything. The format field in your config picks which connector to use. A connector is a small Python class that knows how to read from or write to one specific place (a database, a REST API, a cloud bucket). Built-ins include hive, jdbc, delta, s3, unity, and rest_api. Need a new data source? Write one and register it — Ubunye discovers plugins automatically.

Folders as architecture. Pipelines are organized as project / use_case / pipeline / task. The CLI uses this structure for scaffolding, execution, and discovery:

pipelines/
  fraud_detection/
    ingestion/
      claim_etl/
      policy_etl/
    feature_engineering/
      claim_features/
    risk_scoring/
      train_model/
      score_claims/

What Can You Build With It

ETL pipelines — move data between Hive, JDBC databases, Delta Lake, S3, REST APIs. Config-driven, scheduled, reproducible.

ML training and inference — define your model behind a simple contract, let the engine handle versioning, storage, and deployment.

RAG document pipelines — ingest documents, extract text, chunk, compute embeddings, load into a vector store. All from YAML.

Feature engineering — compute features once, write to a shared table, reuse across use cases.

Data drift detection — monitor feature distributions between runs, flag when things shift.

Check out the Patterns section in our docs for full examples.

Examples

All worked examples live in ubunye-examples. They install the engine from PyPI, so what you run there is what you get from pip install.

They are grouped by where they run. The same task folder is used in every environment. Only environment variables change.

Databricks (free workspace is enough)

Local machine, Docker, and Kubernetes (no cloud account needed)

AWS and GCP

Submit scripts and CI jobs exist for EMR Serverless and Dataproc Serverless. They are written and documented but need a cloud account to run, and the CI jobs say plainly when they were skipped rather than run.

Connectors

Want to add one? See the plugin guide.

Run Anywhere

The same task folder runs on every environment below. This is tested, not claimed: a CI job runs one task on each and fails the build if the output hashes differ.

One rule makes this work: the task never chooses its own cluster. Which Spark master to use belongs to whoever launches the job, so leave spark.master out of your config. On a laptop the runner sets local[*]. On a cloud the platform sets it, and the engine refuses a config that tries to override it, because a silent single-node run on paid compute is worse than an error.

---------------------------------------|-----------------------------------------------| | Your laptop | spark.master: "local[*]" | | Hadoop / YARN cluster | spark.master: "yarn" | | Kubernetes | spark.master: "k8s://..." | | Databricks notebooks or jobs | Call ubunye.run_task() from Python — Ubunye picks up the active session | | AWS EMR | Runs as an EMR Step |

Don't recognise some of these? That's fine — you only need one. If you're starting out, local[*] runs Spark on your own machine with no setup.

Jinja Templating

Anywhere a string appears in your YAML, you can plug in a variable using {{ … }} syntax (this is called Jinja templating). That's how you keep secrets out of your config, change paths per environment, and inject the run date from the CLI:

# Environment variables
password: "{{ env.DB_PASSWORD }}"

# CLI variables (--var ds=2025-01-01)
path: "s3a://bucket/{{ ds }}/"

# Defaults
path: "s3a://bucket/{{ ds | default('2025-01-01') }}/"

CLI

ubunye init     -d ./pipelines -u <use_case> -p <pipeline> -t <task>   # scaffold
ubunye validate -d ./pipelines -u <use_case> -p <pipeline> -t <task>   # check config
ubunye plan     -d ./pipelines -u <use_case> -p <pipeline> -t <task>   # preview plan
ubunye run      -d ./pipelines -u <use_case> -p <pipeline> -t <task>   # execute
ubunye test run -d ./pipelines -u <use_case> -p <pipeline> -t <task>   # test mode
ubunye lineage list -d ./pipelines -u <use_case> -p <pipeline> -t <task>  # run history
ubunye models list -u <use_case> -m <model> -s <store>                 # model versions

Python API

import ubunye

# Run from Databricks or any Python environment
outputs = ubunye.run_task(task_dir="./pipelines/...", mode="DEV", dt="2024-06-01")

# Multiple tasks
results = ubunye.run_pipeline(
    usecase_dir="./pipelines", usecase="fraud", package="etl",
    tasks=["claim_etl", "features"], mode="DEV",
)

What Ubunye Is Not

It's not an agent framework — use LangChain or CrewAI for that. It's not an orchestrator — use Airflow, Prefect, or Dagster. It's not a compute engine — it runs on Spark.

Ubunye is the standardization layer between your data sources and your applications. It makes the plumbing boring so you can focus on what matters.

Roadmap

• Config-driven ETL pipelines
• Multi-environment profiles
• Jinja templating
• Plugin-based connectors
• CLI scaffolding and execution
• Pydantic config validation
• ML model contract
• Model registry with versioning
• Lineage tracking
• Python API for Databricks
• Databricks Asset Bundles deployment
• Dev notebook scaffolding
• Data drift detection
• ubunye deploy CLI command

Get Involved

We'd love your help. Whether it's a new connector, a bug fix, a typo, or just telling us what you're building — all contributions matter.

• 🐛 Report a bug
• 💡 Request a feature
• 📖 Read the contributing guide
• ⭐ Star the repo if you find it useful — it helps more than you'd think

License

MIT License

Built with 🇿🇦 by Ubunye AI Ecosystems