Odibi

Declarative data pipelines. YAML in, star schemas out.

Note: Personal open-source project. See IP_NOTICE.md for details.

CI PyPI Python 3.9+ License Docs

Odibi is a framework for building data pipelines. You describe what you want in YAML; Odibi handles how. Every run generates a "Data Story" — an audit report showing exactly what happened to your data.

🤖 AI/LLM Users: For comprehensive context, see docs/ODIBI_DEEP_CONTEXT.md — 2,200+ lines covering all patterns, transformers, validation, connections, and runtime behavior.

🎯 Try Odibi in 5 Minutes (No Install Needed)

Open In Colab

Click the badge above → run 3 cells → see your first simulation. No Python install, no cloning, no setup.

The notebook walks you through:

  1. pip install odibi (runs in the cloud)
  2. Define a simulation in YAML (sensors, sales data, or industrial equipment)
  3. Run the pipeline → see the output → chart it with Altair

When you're ready for more: 38 simulation configs covering buildings, compressors, reactors, cooling towers, wastewater, production lines, and sales pipelines.

⚡ Quick Start (Local)

pip install odibi

Option 1: Simulate data from YAML

Create sim.yaml:

project: my_first_sim
engine: pandas
connections:
  output:
    type: local
    base_path: ./odibi-quickstart-output
story:
  connection: output
  path: stories/
system:
  connection: output
pipelines:
  - pipeline: demo
    nodes:
      - name: sensors
        read:
          connection: null
          format: simulation
          options:
            simulation:
              scope:
                start_time: "2026-01-01T00:00:00Z"
                timestep: "5m"
                row_count: 100
                seed: 42
              entities:
                count: 3
                id_prefix: "sensor_"
              columns:
                - name: sensor_id
                  data_type: string
                  generator: {type: constant, value: "{entity_id}"}
                - name: timestamp
                  data_type: timestamp
                  generator: {type: timestamp}
                - name: temperature
                  data_type: float
                  generator:
                    type: random_walk
                    start: 22.0
                    min: 16.0
                    max: 30.0
                    volatility: 0.3
                    mean_reversion: 0.15
        write:
          connection: output
          format: parquet
          path: bronze/sensors.parquet
          mode: overwrite

Validate, plan, then run it. Continue only when the planner response has status == "planned":

odibi validate sim.yaml --format json
cat sim.yaml | odibi plan --stdin --format json

Effects before you run: relative to the directory containing sim.yaml, execution overwrites odibi-quickstart-output/bronze/sensors.parquet; creates timestamped odibi-quickstart-output/stories/demo/YYYY-MM-DD/run_HH-MM-SS.{html,json} Stories; and initializes or updates odibi-quickstart-output/_odibi_system/ meta_* catalog assets. A repeat replaces the sensor target, normally creates a later Story pair subject to retention, and updates run/catalog metadata. To clean up, delete only odibi-quickstart-output/ and the sim.yaml created for this example; do not do so if either path was repurposed.

Run:

python -c "from odibi.pipeline import PipelineManager; PipelineManager.from_yaml('sim.yaml').run()"

Output: odibi-quickstart-output/bronze/sensors.parquet contains simulated sensor data with memory, drift, and mean reversion. No database needed.

Option 2: Build a star schema from CSV

Effects before init: odibi init creates my_project/; it replaces an existing directory only when explicitly forced, and this command does not use --force.

odibi init my_project --template star-schema
cd my_project

Effects before you run: execution overwrites data/gold/dim_customer.parquet, dim_product.parquet, dim_date.parquet, and fact_sales.parquet; writes timestamped Stories below data/gold/stories/{build_dimensions|build_facts}/YYYY-MM-DD/run_HH-MM-SS.{html,json}; and initializes or updates data/gold/_system/meta_*. Repeats replace the four data targets, create later Stories, and update catalog metadata. The generated sample_data/ is inside the project. Cleanup is removal of the newly generated my_project/ only; never delete a reused project.

odibi run odibi.yaml
odibi story last          # View the audit report

Option 3: Clone the reference example

Review the canonical reference's exact effects, repeat behavior, and cleanup before execution.

git clone https://github.com/henryodibi11/Odibi.git
cd Odibi/docs/examples/canonical/runnable
odibi run 04_fact_table.yaml

This builds a complete star schema in seconds:

  • 3 dimension tables (customer, product, date)
  • 1 fact table with FK lookups and orphan handling
  • HTML audit report

See the full breakdown →

📖 The Canonical Example

pipelines:
  - pipeline: build_dimensions
    nodes:
      - name: dim_customer
        read:
          connection: source
          format: csv
          path: customers.csv
        pattern:
          type: dimension
          params:
            natural_key: customer_id
            surrogate_key: customer_sk
            scd_type: 1
        write:
          connection: gold
          format: parquet
          path: dim_customer

      - name: dim_date
        pattern:
          type: date_dimension
          params:
            start_date: "2025-01-01"
            end_date: "2025-12-31"
        write:
          connection: gold
          format: parquet
          path: dim_date

  - pipeline: build_facts
    nodes:
      - name: fact_sales
        depends_on: [dim_customer, dim_date]
        read:
          connection: source
          format: csv
          path: orders.csv
        pattern:
          type: fact
          params:
            grain: [order_id, line_item_id]
            dimensions:
              - source_column: customer_id
                dimension_table: dim_customer
                dimension_key: customer_id
                surrogate_key: customer_sk
            orphan_handling: unknown
        write:
          connection: gold
          format: parquet
          path: fact_sales

Full runnable example →

🚀 Key Features

Feature Description
Data Stories Every run generates an HTML audit report
Dimensional Patterns 6 built-in patterns: SCD1/SCD2, date dimension, fact tables, merge, aggregation
56 Transformers Comprehensive library for data manipulation and quality
Validation & Contracts Fail-fast checks, quarantine bad rows
Multi-Engine Pandas, Polars, and Spark — same config across all engines
Production Ready Retry, alerting, secrets, Delta Lake support
Battle-Tested 5500+ tests ensure reliability and correctness

📚 Documentation

Goal Link
Get running in 10 minutes Golden Path
Copy THE working example THE_REFERENCE.md
Solve a specific problem Playbook
Understand when to use what Decision Guide
See all config options YAML Schema

📦 Installation

# Standard (Pandas engine)
pip install odibi

# With Polars engine
pip install "odibi[polars]"

# With Spark + Azure support
pip install "odibi[spark,azure]"

# All engines and features
pip install "odibi[all]"

🎯 Who is this for?

  • Solo data engineers building pipelines without a team
  • Analytics engineers moving from dbt to Python-based pipelines
  • Anyone tired of writing the same boilerplate for every project

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md.

Maintainer: Henry Odibi (@henryodibi11)
License: Apache 2.0

Project links

Data verified by PyPI on Aug 11, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for odibiengineering from gravatar.com odibiengineering

Credits

PyPI data
Data sourced directly from PyPI's database.

Author: Henry Odibi

GitHub Statistics

Data verified by PyPI on Aug 11, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

Apache Software License (Apache-2.0)

Requires

Python >=3.9

Provides Extra

cli lineage telemetry spark pandas azure sql postgres polars thermodynamics mcp mcp-rag all dev

Tags

data-engineering pipeline etl workflow orchestration

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

odibi-3.15.0.tar.gz (2.0 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

odibi-3.15.0-py3-none-any.whl (2.2 MB view details)

Uploaded Python 3

File details

Details for the file odibi-3.15.0.tar.gz.

File metadata

  • Download URL: odibi-3.15.0.tar.gz
  • Upload date:
  • Size: 2.0 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for odibi-3.15.0.tar.gz
Algorithm Hash digest
SHA256 483e138e079786fb544cbb6f4c2a77c2a56d9de46981cac4c99d47fea9eea53b
MD5 5627f52b9444ad09944afcde6a31e947
BLAKE2b-256 0823b477b58d6d36e018e0e019a5cd10750675f8f4e09eeedb7fa3ec87f8d7ac

See more details on using hashes here.

Provenance

The following attestation bundles were made for odibi-3.15.0.tar.gz:

Publisher: publish.yml on henryodibi11/Odibi

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file odibi-3.15.0-py3-none-any.whl.

File metadata

  • Download URL: odibi-3.15.0-py3-none-any.whl
  • Upload date:
  • Size: 2.2 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for odibi-3.15.0-py3-none-any.whl
Algorithm Hash digest
SHA256 0d4e9429529138060ba2ca031ee14a322ced56627d4574c7e97d80c01766c8eb
MD5 6ca2fd08fc7ef50975e5c8e2be7190f2
BLAKE2b-256 8c0a08d0235b3f6d3d782042612128989afb088cfcb241ac005c3607063e3278

See more details on using hashes here.

Provenance

The following attestation bundles were made for odibi-3.15.0-py3-none-any.whl:

Publisher: publish.yml on henryodibi11/Odibi

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

3.15.0 This release

2 files

3.14.0

2 files

3.13.8

2 files

3.13.7

2 files

3.13.6

2 files

3.13.5

2 files

3.13.4

2 files

3.13.3

2 files

3.13.2

2 files

3.13.1

2 files

3.13.0

2 files

3.12.1

2 files

3.12.0

2 files

3.11.0

2 files

3.10.0

2 files

3.9.0

2 files

3.8.2

2 files

3.8.1

2 files

3.8.0

2 files

3.7.4

2 files

3.7.3

2 files

3.7.2

2 files

3.7.1

2 files

3.7.0

2 files

3.6.3

2 files

3.6.2

2 files

3.6.1

2 files

3.6.0

2 files

3.5.0

2 files

3.4.7

2 files

3.4.6

2 files

3.4.5

2 files

3.4.4

2 files

3.4.3

2 files

3.4.2

2 files

3.4.1

2 files

3.4.0

2 files

3.3.0

2 files

2.23.0

2 files

2.22.1

2 files

2.22.0

2 files

2.21.0

2 files

2.20.1

2 files

2.20.0

2 files

2.18.0

2 files

2.17.0

2 files

2.16.0

2 files

2.15.3

2 files

2.15.2

2 files

2.15.1

2 files

2.15.0

2 files

2.14.0

2 files

2.13.1

2 files

2.12.0

2 files

2.11.2

2 files

2.11.1

2 files

2.11.0

2 files

2.10.0

2 files

2.9.0

2 files

2.8.0

2 files

2.7.0

2 files

2.6.6

2 files

2.6.5

2 files

2.6.4

2 files

2.6.3

2 files

2.6.2

2 files

2.6.1

2 files

2.5.0

2 files

2.4.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page