odibi 3.4.4

A declarative data engineering framework - Explicit over implicit, Stories over magic

Odibi

Declarative data pipelines. YAML in, star schemas out.

Note: Personal open-source project. See IP_NOTICE.md for details.

Odibi is a framework for building data pipelines. You describe what you want in YAML; Odibi handles how. Every run generates a "Data Story" — an audit report showing exactly what happened to your data.

🤖 AI/LLM Users: For comprehensive context, see docs/ODIBI_DEEP_CONTEXT.md — 2,200+ lines covering all patterns, transformers, validation, connections, and runtime behavior.

---

⚡ Quick Start

pip install odibi

Option 1: Start from a template

odibi init my_project --template star-schema
cd my_project
odibi run odibi.yaml
odibi story last          # View the audit report

Option 2: Clone the reference example

git clone https://github.com/henryodibi11/Odibi.git
cd Odibi/docs/examples/canonical/runnable
odibi run 04_fact_table.yaml

This builds a complete star schema in seconds:

• 3 dimension tables (customer, product, date)
• 1 fact table with FK lookups and orphan handling
• HTML audit report

See the full breakdown →

---

📖 The Canonical Example

pipelines:
  - pipeline: build_dimensions
    nodes:
      - name: dim_customer
        read:
          connection: source
          format: csv
          path: customers.csv
        pattern:
          type: dimension
          params:
            natural_key: customer_id
            surrogate_key: customer_sk
            scd_type: 1
        write:
          connection: gold
          format: parquet
          path: dim_customer

      - name: dim_date
        pattern:
          type: date_dimension
          params:
            start_date: "2025-01-01"
            end_date: "2025-12-31"
        write:
          connection: gold
          format: parquet
          path: dim_date

  - pipeline: build_facts
    nodes:
      - name: fact_sales
        depends_on: [dim_customer, dim_date]
        read:
          connection: source
          format: csv
          path: orders.csv
        pattern:
          type: fact
          params:
            grain: [order_id, line_item_id]
            dimensions:
              - source_column: customer_id
                dimension_table: dim_customer
                dimension_key: customer_id
                surrogate_key: customer_sk
            orphan_handling: unknown
        write:
          connection: gold
          format: parquet
          path: fact_sales

Full runnable example →

---

🚀 Key Features

Feature	Description
Data Stories	Every run generates an HTML audit report
Dimensional Patterns	6 built-in patterns: SCD1/SCD2, date dimension, fact tables, merge, aggregation
56 Transformers	Comprehensive library for data manipulation and quality
Validation & Contracts	Fail-fast checks, quarantine bad rows
Multi-Engine	Pandas, Polars, and Spark — same config across all engines
Production Ready	Retry, alerting, secrets, Delta Lake support
Battle-Tested	5500+ tests ensure reliability and correctness

---

📚 Documentation

Goal	Link
Get running in 10 minutes	Golden Path
Copy THE working example	THE_REFERENCE.md
Solve a specific problem	Playbook
Understand when to use what	Decision Guide
See all config options	YAML Schema

---

📦 Installation

# Standard (Pandas engine)
pip install odibi

# With Polars engine
pip install "odibi[polars]"

# With Spark + Azure support
pip install "odibi[spark,azure]"

# All engines and features
pip install "odibi[all]"

---

🎯 Who is this for?

• Solo data engineers building pipelines without a team
• Analytics engineers moving from dbt to Python-based pipelines
• Anyone tired of writing the same boilerplate for every project

---

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md.

---

Maintainer: Henry Odibi (@henryodibi11)
License: Apache 2.0