Odibi

Declarative data pipelines. YAML in, star schemas out.

Note: Personal open-source project. See IP_NOTICE.md for details.

Odibi is a framework for building data pipelines. You describe what you want in YAML; Odibi handles how. Every run generates a "Data Story" — an audit report showing exactly what happened to your data.

🤖 AI/LLM Users: For comprehensive context, see docs/ODIBI_DEEP_CONTEXT.md — 2,200+ lines covering all patterns, transformers, validation, connections, and runtime behavior.

🎯 Try Odibi in 5 Minutes (No Install Needed)

Click the badge above → run 3 cells → see your first simulation. No Python install, no cloning, no setup.

The notebook walks you through:

1. pip install odibi (runs in the cloud)
2. Define a simulation in YAML (sensors, sales data, or industrial equipment)
3. Run the pipeline → see the output → chart it with Altair

When you're ready for more: 38 simulation configs covering buildings, compressors, reactors, cooling towers, wastewater, production lines, and sales pipelines.

⚡ Quick Start (Local)

pip install odibi

Option 1: Simulate data from YAML

Create sim.yaml:

project: my_first_sim
engine: pandas
connections:
  output:
    type: local
    base_path: ./odibi-quickstart-output
story:
  connection: output
  path: stories/
system:
  connection: output
pipelines:
  - pipeline: demo
    nodes:
      - name: sensors
        read:
          connection: null
          format: simulation
          options:
            simulation:
              scope:
                start_time: "2026-01-01T00:00:00Z"
                timestep: "5m"
                row_count: 100
                seed: 42
              entities:
                count: 3
                id_prefix: "sensor_"
              columns:
                - name: sensor_id
                  data_type: string
                  generator: {type: constant, value: "{entity_id}"}
                - name: timestamp
                  data_type: timestamp
                  generator: {type: timestamp}
                - name: temperature
                  data_type: float
                  generator:
                    type: random_walk
                    start: 22.0
                    min: 16.0
                    max: 30.0
                    volatility: 0.3
                    mean_reversion: 0.15
        write:
          connection: output
          format: parquet
          path: bronze/sensors.parquet
          mode: overwrite

Validate, plan, then run it. Continue only when the planner response has status == "planned":

odibi validate sim.yaml --format json
cat sim.yaml | odibi plan --stdin --format json

Effects before you run: relative to the directory containing sim.yaml, execution overwrites odibi-quickstart-output/bronze/sensors.parquet; creates timestamped odibi-quickstart-output/stories/demo/YYYY-MM-DD/run_HH-MM-SS.{html,json} Stories; and initializes or updates odibi-quickstart-output/_odibi_system/ meta_* catalog assets. A repeat replaces the sensor target, normally creates a later Story pair subject to retention, and updates run/catalog metadata. To clean up, delete only odibi-quickstart-output/ and the sim.yaml created for this example; do not do so if either path was repurposed.

Run:

python -c "from odibi.pipeline import PipelineManager; PipelineManager.from_yaml('sim.yaml').run()"

Output: odibi-quickstart-output/bronze/sensors.parquet contains simulated sensor data with memory, drift, and mean reversion. No database needed.

Option 2: Build a star schema from CSV

Effects before init: odibi init creates my_project/; it replaces an existing directory only when explicitly forced, and this command does not use --force.

odibi init my_project --template star-schema
cd my_project

Effects before you run: execution overwrites data/gold/dim_customer.parquet, dim_product.parquet, dim_date.parquet, and fact_sales.parquet; writes timestamped Stories below data/gold/stories/{build_dimensions|build_facts}/YYYY-MM-DD/run_HH-MM-SS.{html,json}; and initializes or updates data/gold/_system/meta_*. Repeats replace the four data targets, create later Stories, and update catalog metadata. The generated sample_data/ is inside the project. Cleanup is removal of the newly generated my_project/ only; never delete a reused project.

odibi run odibi.yaml
odibi story last          # View the audit report

Option 3: Clone the reference example

Review the canonical reference's exact effects, repeat behavior, and cleanup before execution.

git clone https://github.com/henryodibi11/Odibi.git
cd Odibi/docs/examples/canonical/runnable
odibi run 04_fact_table.yaml

This builds a complete star schema in seconds:

• 3 dimension tables (customer, product, date)
• 1 fact table with FK lookups and orphan handling
• HTML audit report

See the full breakdown →

📖 The Canonical Example

pipelines:
  - pipeline: build_dimensions
    nodes:
      - name: dim_customer
        read:
          connection: source
          format: csv
          path: customers.csv
        pattern:
          type: dimension
          params:
            natural_key: customer_id
            surrogate_key: customer_sk
            scd_type: 1
        write:
          connection: gold
          format: parquet
          path: dim_customer

      - name: dim_date
        pattern:
          type: date_dimension
          params:
            start_date: "2025-01-01"
            end_date: "2025-12-31"
        write:
          connection: gold
          format: parquet
          path: dim_date

  - pipeline: build_facts
    nodes:
      - name: fact_sales
        depends_on: [dim_customer, dim_date]
        read:
          connection: source
          format: csv
          path: orders.csv
        pattern:
          type: fact
          params:
            grain: [order_id, line_item_id]
            dimensions:
              - source_column: customer_id
                dimension_table: dim_customer
                dimension_key: customer_id
                surrogate_key: customer_sk
            orphan_handling: unknown
        write:
          connection: gold
          format: parquet
          path: fact_sales

Full runnable example →

🚀 Key Features

📚 Documentation

📦 Installation

# Standard (Pandas engine)
pip install odibi

# With Polars engine
pip install "odibi[polars]"

# With Spark + Azure support
pip install "odibi[spark,azure]"

# All engines and features
pip install "odibi[all]"

🎯 Who is this for?

• Solo data engineers building pipelines without a team
• Analytics engineers moving from dbt to Python-based pipelines
• Anyone tired of writing the same boilerplate for every project

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md.

Maintainer: Henry Odibi (@henryodibi11)
License: Apache 2.0