data-product-forge 0.10.1

FLUID Build System: plan/apply/visualize data products across providers (GCP, Snowflake, Local) with policy & contract tests.

Stop writing boilerplate. Start declaring Data Products.

The declarative control plane for data engineering in the Agentic Era.

Documentation · Getting Started · FLUID Specification · The Book · Community

_{Install through deploy in 30 seconds, against the local DuckDB provider. Click through to the demos library for 8 more casts (GCP, AWS, Snowflake, agentPolicy, AI copilot…).}

---

🚀 Latest: v0.8.6 on PyPI · contract schema fluidVersion 0.7.4 · 📖 Full documentation →

Jump to: Why FLUID Forge? · What is FLUID? · Why We Built This · 60 Seconds to Magic · Installation · Contract Anatomy · Providers · CLI Reference · Templates · Contributing

Why FLUID Forge?

1 file. 4 clouds. 0 rewrites. One contract.fluid.yaml. Swap binding.platform and ship.

Validate → deploy in 30s. Local DuckDB out of the box. No cloud account, no credit card.

Compiles to native IAM. policy-compile emits real BigQuery/Snowflake/AWS bindings.

Agentic governance, enforced. agentPolicy gates which LLMs can read which fields, via the Fluid MCP output port (fluid mcp output-port serve).

---

🌊 What is FLUID?

FLUID (Federated Layered Universal Instructional Declaration) is an open-source declarative framework for building, validating, and deploying data products across any cloud. You write a single YAML contract describing what your data product is — its schema, quality rules, access policies, and scheduling — and FLUID Forge compiles that into provider-specific infrastructure (BigQuery, Snowflake, AWS Glue, Athena, or local DuckDB) with full governance baked in.

Think of it as Terraform for data products: one contract, many clouds, zero boilerplate.

The everyday flow is three commands — validate, plan, apply. Behind the scenes, FLUID Forge ships an 11-stage deterministic pipeline with cryptographic plan-binding and per-stage hard gates:

bundle → validate → generate-artifacts → validate-artifacts → diff →
plan → apply → policy-apply → verify → publish → schedule-sync

You rarely run all 11 by hand; fluid generate ci emits a parameterized CI pipeline (Jenkins, GitHub Actions, GitLab CI, Azure DevOps, Bitbucket, CircleCI, Tekton) that runs every stage in order with cryptographic binding between stages 6 and 7 — apply refuses to proceed if the plan has been tampered with since it was computed. See AGENTS.md for the full stage lifecycle and the apply mode matrix (dry-run, create-only, amend, amend-and-build, replace, replace-and-build).

Under the hood (cloud providers): OpenTofu autogen. fluid apply compiles the contract to a deterministic OpenTofu main.tf.json and delegates apply / state / drift / idempotency to the tofu binary — battle-tested infrastructure code instead of hand-rolled per-cloud apply. The emitter is modular: one IacProviderPlugin per cloud (fluid_build/iac/providers/{aws,gcp,snowflake}.py, dbt-adapter pattern). Preview the emitted module before applying with fluid generate iac <contract>. See AUTOGEN_SPIKE.md for the architecture and HONESTLY_TESTED.md for the coverage matrix (unit / emulator / live cloud).

---

🤯 Why We Built This

Data engineering is stuck in the dark ages of imperative spaghetti code. You want to ship data products fast, but compliance teams demand governance. You end up with Configuration Sprawl: .tf files for Terraform, schema.yml for dbt, .rego for OPA, and a web of Airflow DAGs.

FLUID Forge is the compiler that ends the chaos. You declare what your data product is. The CLI compiles that into a validated, deterministic execution plan across any supported cloud.

The Old Way vs. The FLUID Way

🛑 The Old Way (Imperative Chaos)	✨ The FLUID Way (Declarative Order)
Weeks of boilerplate to wire up IaC, SQL, and DAGs.	Minutes to deploy. Just declare your intent and apply.
Vendor lock-in. Your DAGs only work on one cloud.	Provider-agnostic. Switch clouds by changing one line of YAML.
Governance as an afterthought. Manual compliance tickets.	Governance-as-Code. Policies compile to native IAM before deployment.
"Works on my machine." Broken production deploys.	Deterministic plans. See exactly what will change before it runs.
AI hallucinations. Agents don't understand your tables.	Semantic Truth. Built-in OSI semantics so LLMs query perfectly.

---

⚡ 60 Seconds to Magic

Data engineering shouldn't require weeks of handwritten infrastructure code, bespoke CI/CD pipelines, and copy-pasted SQL.

What Terraform did for infrastructure, FLUID Forge does for data products.

# 1. Install the CLI
pip install data-product-forge

# 2. Validate your data product contract
fluid validate contract.fluid.yaml

# 3. See exactly what will happen
fluid plan contract.fluid.yaml

# 4. Deploy infrastructure, logic, and governance — instantly
fluid apply contract.fluid.yaml

That's it. You just deployed a versioned, governed, and orchestrated Data Product from a single YAML file.

Want to move from local to Google Cloud? pip install "data-product-forge[gcp]" → change platform: local to platform: gcp → run fluid apply. Done.

From an existing catalog — fluid forge data-model from-source (V1.5)

Already have your tables registered in Snowflake Horizon, Databricks Unity, BigQuery, Dataplex, AWS Glue, DataHub, or Data Mesh Manager? Forge a Fluid contract directly from the metadata you've already curated — descriptions, tags, lineage, classifications, ownership — instead of re-typing it.

# 1. Install the catalog extra (or use [catalogs] for all of them).
pip install "data-product-forge[snowflake]"

# 2. One-time source setup (saves to ~/.fluid/sources.yaml; secrets to OS keyring).
fluid ai setup --source snowflake --name snowflake-prod

# 3. Forge a Data Vault 2.0 model from a schema.
fluid forge data-model from-source \
  --source snowflake --credential-id snowflake-prod \
  --database <DATABASE> --schema <SCHEMA> \
  --technique data-vault-2 \
  -o customer_orders.fluid.yaml

# 4. Generate dbt transformations from the same contract.
fluid generate speed-transformation customer_orders.fluid.yaml -o ./dbt_customer_orders

Three guarantees that hold across every catalog: read-only metadata access (no SELECT * against any data table), per-call credentials (the MCP server never holds your secrets), and full audit trail (every catalog read writes a redacted event under ~/.fluid/store/audit/).

The same flow is exposed via the MCP forge_from_source tool — Claude Code, Cursor, and any MCP client can drive a catalog forge from inside the editor. See the catalogs walkthrough for per-catalog privilege grants and end-to-end demos.

Which ODPS? — Bitol center stage, LF/ODPI in the back pocket

Two different specs share the ODPS acronym. FLUID treats Bitol Open Data Product Standard v1.0.0 as the default, center-stage target; the Linux Foundation / ODPI Open Data Product Specification v4.1 is a secondary, opt-in export for catalogs that require it.

Standard	Role	What it is	Spec	CLI selector	--format
Bitol ODPS v1.0.0	default	Open Data Product Standard. Bidirectional; product wrapper that references ODCS contracts by contractId.	bitol-io/open-data-product-standard	fluid odps --spec bitol-1.0.0 (default)	--format odps (default) / --format odps-bitol
LF/ODPI ODPS v4.1	opt-in	Open Data Product Specification. Hosted by the Linux Foundation / Open Data Product Initiative (ODPI). Export-only; single JSON document.	Open-Data-Product-Initiative/v4.1	fluid odps --spec odps-4.1	--format odps-v4.1

Bitol export emits 1 ODPS doc + N sibling <contractId>.odcs.yaml fragments; import reverses it — a single ODPS file, a directory bundle, or a lone ODCS file all converge on one validated FLUID contract (and fluid forge --seed-from <path> accepts the same three shapes as an AI-authoring seed).

Back-compat aliases

All emit a WARNING pointing at the canonical form:

• --spec odpi-4.1 and --version 4.1 — historical labels for the LF/ODPI v4.1 spec

→ full ODPS guide

From dev to CI — fluid generate ci

When you're ready to run the full 11-stage pipeline in CI instead of manually, one command emits the pipeline file for your CI system:

# Jenkins (ships the richest template today — every stage parameterized)
fluid generate ci --system jenkins --out Jenkinsfile

# GitHub Actions, GitLab, Azure DevOps, Bitbucket, CircleCI, Tekton
fluid generate ci --system github --out .github/workflows/fluid.yml

The generated Jenkinsfile has a "Build with Parameters" dialog where operators pick the subset of stages to run + the apply mode + the publish targets + schedule-sync scheduler — all without editing Groovy:

Param	Default	Purpose
RUN_STAGE_N_* (11 booleans)	all true	Toggle individual stages
APPLY_MODE	amend	6-mode choice
ALLOW_DATA_LOSS	false	Required for replace* outside dev
PUBLISH_TARGETS	datamesh-manager	Space-separated list
SCHEDULER	`` (none)	DAG push target for stage 11

Generated Jenkinsfiles install fluid via pip install data-product-forge at build time by default. For lab iteration against a local forge-cli checkout, fluid generate ci --install-mode dev-source emits a Jenkinsfile that bind-mounts your checkout inside the Jenkins container — zero pip install, changes reflect live.

---

Installation

FLUID Forge is modular — install only what you need. Requires Python 3.10+.

pip install data-product-forge                # Minimal — CLI + Local/DuckDB provider
pip install "data-product-forge[gcp]"         # + Google Cloud (BigQuery, GCS, Composer)
pip install "data-product-forge[aws]"         # + AWS (S3, Glue, Athena, Redshift)
pip install "data-product-forge[snowflake]"   # + Snowflake
pip install "data-product-forge[all]"         # Everything (all providers + dev tools)

💡 Tip: We recommend pipx for an isolated global install: pipx install "data-product-forge[all]"

Verify, then scaffold your first product:

fluid --version
fluid init demo --template hello-world

Install from source (for contributors)

Choose your platform below for step-by-step instructions.

Prerequisites

Requirement	Version
Python	3.10 or higher
pip	Latest recommended
Git	Any recent version

---

🍎 macOS

1. Install system dependencies

# Install Xcode Command Line Tools (includes Git and make)
xcode-select --install

# Or install Python & Git via Homebrew (recommended)
brew install python git

2. Verify Python

python3 --version   # Should print Python 3.10+

3. Clone and install

git clone https://github.com/Agenticstiger/forge-cli.git
cd forge-cli

# Create a virtual environment (recommended)
python3 -m venv .venv
source .venv/bin/activate

# Install in editable mode with the local provider
pip install --upgrade pip wheel
pip install -e ".[local]"

4. Verify the installation

fluid --version
fluid validate examples/01-hello-world/contract.fluid.yaml

Tip: You can also run the automated setup script:

chmod +x setup.sh && ./setup.sh

🐧 Ubuntu / Debian Linux

1. Install system dependencies

sudo apt update
sudo apt install -y python3 python3-venv python3-pip git build-essential

2. Verify Python

python3 --version   # Should print Python 3.10+

3. Clone and install

git clone https://github.com/Agenticstiger/forge-cli.git
cd forge-cli

# Create a virtual environment (recommended)
python3 -m venv .venv
source .venv/bin/activate

# Install in editable mode with the local provider
pip install --upgrade pip wheel
pip install -e ".[local]"

4. Verify the installation

fluid --version
fluid validate examples/01-hello-world/contract.fluid.yaml

Tip: You can also run the automated setup script:

chmod +x setup.sh && ./setup.sh

🪟 Windows

1. Install Python

Download and install Python 3.10+ from python.org.

Important: Check "Add Python to PATH" during installation.

2. Install Git

Download and install Git from git-scm.com, or install via a package manager:

# Via winget
winget install Git.Git

# Or via Chocolatey
choco install git

3. Verify prerequisites

python --version   # Should print Python 3.10+
git --version

4. Clone and install

PowerShell:

git clone https://github.com/Agenticstiger/forge-cli.git
cd forge-cli

# Create a virtual environment (recommended)
python -m venv .venv
.venv\Scripts\Activate.ps1

# Install in editable mode with the local provider
pip install --upgrade pip wheel
pip install -e ".[local]"

Command Prompt (cmd):

git clone https://github.com/Agenticstiger/forge-cli.git
cd forge-cli

python -m venv .venv
.venv\Scripts\activate.bat

pip install --upgrade pip wheel
pip install -e ".[local]"

5. Verify the installation

fluid --version
fluid validate examples\01-hello-world\contract.fluid.yaml

Tip: You can also run the automated setup script:

• PowerShell: .\setup.ps1
• Command Prompt: setup.bat

Adding a provider to a source checkout? Use the editable form, e.g. pip install -e ".[gcp]" / ".[snowflake]" / ".[all]".

---

🧬 Anatomy of a FLUID Contract

Everything starts with contract.fluid.yaml — the single source of truth for your data product's entire lifecycle.

fluidVersion: "0.7.4"
kind: DataProduct
id: example.customer_360
name: Customer 360
domain: analytics

metadata:
  layer: Gold              # medallion vocabulary
  productType: CDP         # Data Mesh vocabulary — Bronze↔SDP, Silver↔ADP, Gold↔CDP
  owner:
    team: data-platform
    email: [e-mail]

# 1. THE LOGIC — How is it built?
builds:
  - id: transform_customer
    pattern: embedded-logic
    engine: sql
    properties:
      sql: |
        SELECT user_id, email, LTV
        FROM raw.users JOIN raw.orders USING (user_id)

# 2. THE INTERFACE — What does it output?
exposes:
  - exposeId: customer_profiles
    kind: table
    binding:
      platform: snowflake              # ← Change to 'gcp' or 'aws' instantly
      format: snowflake_table
      location:
        database: PROD
        schema: GOLD
        table: CUST_360
    contract:
      schema:
        - name: email
          type: string
          sensitivity: pii             # ← Triggers auto-masking/encryption

# 3. THE GOVERNANCE — Who (or what) can access it?
accessPolicy:
  grants:
    - principal: "group:[e-mail]"
      permissions: ["read"]

agentPolicy:                           # ← Agentic Era Governance
  allowedModels: ["gpt-4", "claude-3"]
  allowedUseCases: ["analysis", "summarization"]

---

🔌 Providers — Bring Your Own Cloud

Providers are the bridge between your declarative contract and your target execution environment.

Provider	Target Ecosystem	Superpowers
💻 local	DuckDB, Local FS	Zero-config. Runs anywhere. Perfect for dev/test.
☁️ gcp	Google Cloud	BigQuery, GCS, Composer (Airflow), Dataform, IAM.
🌩️ aws	Amazon Web Services	S3, Glue, Athena, Redshift, MWAA, IAM.
❄️ snowflake	Snowflake	Databases, schemas, streams, tasks, RBAC, sharing.

Open-data-standard export formats (not cloud providers): ODPS (Open Data Product Standard — Bitol / LF-ODPI) via fluid odps, plus odcs and datamesh-manager integrations.

---

💻 CLI Command Reference

FLUID Forge is designed to feel as natural as git or terraform.

Core Lifecycle

fluid init                           # Scaffold a new Data Product contract
fluid validate contract.fluid.yaml   # Validate schema, dependencies, syntax
fluid plan contract.fluid.yaml       # Generate a deterministic execution plan
fluid apply contract.fluid.yaml      # Execute the plan against your target provider
fluid verify contract.fluid.yaml     # Post-deployment data quality & compliance checks
fluid ship contract.fluid.yaml       # One-shot macro: validate → bundle → plan → apply

AI & Code Generation

fluid forge                                  # 🤖 Interactive, AI-powered project creation
fluid generate-airflow contract.fluid.yaml   # Compile contract → native Airflow DAG
fluid generate-pipeline contract.fluid.yaml  # Scaffold transformation code
fluid generate iac contract.fluid.yaml       # Emit a deterministic OpenTofu main.tf.json
fluid stats                                  # Aggregate LLM cost across forge runs
fluid agents list                            # List / show / prune forge runs (.fluid/agents/)

Staged Forge Pipeline (v1.0)

# Forge a reviewable data-model contract from an intent or DDL files.
# OSI v0.1.1 semantics + Logical IR sidecar are emitted alongside the contract.
fluid forge data-model from-intent intent.yaml -o customer_orders.fluid.yaml
fluid forge data-model from-ddl --ddl legacy/*.sql -o customer_orders.fluid.yaml \
    --source-type snowflake --technique data-vault-2

# Validate, diff, dump-DDL companions
fluid forge data-model validate customer_orders.fluid.yaml
fluid forge data-model diff old.model.json new.model.json
fluid forge data-model dump-ddl --database <DATABASE> --schema <SCHEMA> -o /tmp/snapshot.sql

# Memory + roadmap
fluid memory show project|team|episodic|semantic
fluid memory search "<query>" --ns memory/semantic --mode hybrid
fluid roadmap

# MCP server (Claude Code, Cursor, …)
fluid mcp serve

Detailed usage: docs/forge-data-model.md. Provider matrix + setup: docs/PROVIDERS.md. OSI semantics shape: docs/OSI.md. Migration from pre-v1.0: docs/MIGRATION.md.

Governance & Compliance

fluid policy-compile contract.fluid.yaml   # Translate policies → native IAM
fluid contract-tests contract.fluid.yaml   # Run assertion suites

MCP output port — serve data products to AI agents

Turn any FLUID expose into an agent-consumable MCP server with the contract's agentPolicy enforced at every read.

# stdio (default — for Claude Desktop / Cursor / MCP Inspector)
fluid mcp output-port serve contract.fluid.yaml

# HTTP / SSE for network deployments (front with mTLS proxy)
FLUID_MCP_AUTH_TOKEN=$(openssl rand -hex 32) \
  fluid mcp output-port serve contract.fluid.yaml \
  --transport http --host 0.0.0.0 --port 8765

# CLI overrides for ops / incident response
fluid mcp output-port serve contract.fluid.yaml \
  --allow-models claude-haiku-4-5-20251001 \
  --deny-use-cases training,fine_tuning

Drivers in-tree: DuckDB · Postgres · Snowflake · BigQuery · AWS Athena. See examples/mcp-output-port-docker/ for a one-command Postgres demo + the proxy/ templates (Caddy / nginx) for production mTLS.

Visualization

fluid graph contract.fluid.yaml   # Graphviz DAG of internal lineage
fluid docs contract.fluid.yaml    # Auto-generate documentation from contract

Diagnostics

fluid doctor   # Environment diagnostics — Python, providers, CLIs, creds

---

🎓 Templates

Don't start from scratch. fluid init ships with battle-tested enterprise patterns:

fluid init --template customer-360

Template	What You Get
hello-world	The basics — start here
incremental-processing	Append/Merge load patterns
multi-source	Complex DAG dependency orchestration
policy-examples	Advanced RBAC and AI agent governance

---

🤝 Contributing

FLUID Forge is community-driven. We want your ideas, providers, and pull requests!

1. Fork the repo
2. Run make setup for a full dev environment
3. Check out CONTRIBUTING.md for style guides and architecture overview

Please read our Code of Conduct before participating.

To report security vulnerabilities, see SECURITY.md.

---

License

Apache License 2.0 · Copyright 2024–2026 Agentics Transformation Pty Ltd

---

Built for the future of data. Built for the Agentic Era.

Documentation · FLUID Specification · The Book · PyPI · Issues

---

🇿🇦 Proudly developed by dustlabs.co.za