fluid-forge 0.7.9

Stop writing boilerplate. Start declaring Data Products.

The declarative control plane for data engineering in the Agentic Era.

Documentation · Getting Started · FLUID Specification · The Book · Community

🌊 What is FLUID?

FLUID (Federated Layered Universal Instructional Declaration) is an open-source declarative framework for building, validating, and deploying data products across any cloud. You write a single YAML contract describing what your data product is — its schema, quality rules, access policies, and scheduling — and FLUID Forge compiles that into provider-specific infrastructure (BigQuery, Snowflake, AWS Glue, Athena, or local DuckDB) with full governance baked in.

Think of it as Terraform for data products: one contract, many clouds, zero boilerplate.

contract.fluid.yaml  →  fluid validate  →  fluid plan  →  fluid apply
       (declare)          (check)           (preview)       (deploy)

🤯 Why We Built This

Data engineering is stuck in the dark ages of imperative spaghetti code. You want to ship data products fast, but compliance teams demand governance. You end up with Configuration Sprawl: .tf files for Terraform, schema.yml for dbt, .rego for OPA, and a web of Airflow DAGs.

FLUID Forge is the compiler that ends the chaos. You declare what your data product is. The CLI compiles that into a validated, deterministic execution plan across any supported cloud.

The Old Way vs. The FLUID Way

⚡ 60 Seconds to Magic

Data engineering shouldn't require weeks of handwritten infrastructure code, bespoke CI/CD pipelines, and copy-pasted SQL.

What Terraform did for infrastructure, FLUID Forge does for data products.

# 1. Install the CLI
pip install fluid-forge

# 2. Validate your data product contract
fluid validate contract.fluid.yaml

# 3. See exactly what will happen
fluid plan contract.fluid.yaml

# 4. Deploy infrastructure, logic, and governance — instantly
fluid apply contract.fluid.yaml

That's it. You just deployed a versioned, governed, and orchestrated Data Product from a single YAML file.

Want to move from local to Google Cloud? pip install "fluid-forge[gcp]" → change platform: local to platform: gcp → run fluid apply. Done.

🖥️ First-Time Setup from Source

Choose your platform below for step-by-step instructions.

Prerequisites

# Install Xcode Command Line Tools (includes Git and make)
xcode-select --install

# Or install Python & Git via Homebrew (recommended)
brew install python git

python3 --version   # Should print Python 3.9+

git clone https://github.com/Agentics-Rising/forge-cli.git
cd forge-cli

# Create a virtual environment (recommended)
python3 -m venv .venv
source .venv/bin/activate

# Install in editable mode with the local provider
pip install --upgrade pip wheel
pip install -e ".[local]"

fluid --version
fluid validate examples/01-hello-world/contract.fluid.yaml

sudo apt update
sudo apt install -y python3 python3-venv python3-pip git build-essential

python3 --version   # Should print Python 3.9+

git clone https://github.com/Agentics-Rising/forge-cli.git
cd forge-cli

# Create a virtual environment (recommended)
python3 -m venv .venv
source .venv/bin/activate

# Install in editable mode with the local provider
pip install --upgrade pip wheel
pip install -e ".[local]"

fluid --version
fluid validate examples/01-hello-world/contract.fluid.yaml

# Via winget
winget install Git.Git

# Or via Chocolatey
choco install git

python --version   # Should print Python 3.9+
git --version

git clone https://github.com/Agentics-Rising/forge-cli.git
cd forge-cli

# Create a virtual environment (recommended)
python -m venv .venv
.venv\Scripts\Activate.ps1

# Install in editable mode with the local provider
pip install --upgrade pip wheel
pip install -e ".[local]"

git clone https://github.com/Agentics-Rising/forge-cli.git
cd forge-cli

python -m venv .venv
.venv\Scripts\activate.bat

pip install --upgrade pip wheel
pip install -e ".[local]"

fluid --version
fluid validate examples\01-hello-world\contract.fluid.yaml

Installing additional providers

Once the base install is working, add cloud providers as needed:

pip install -e ".[gcp]"         # + Google Cloud (BigQuery, GCS, Composer)
pip install -e ".[snowflake]"   # + Snowflake
pip install -e ".[all]"         # Everything (all providers + dev tools)

🧬 Anatomy of a FLUID Contract

Everything starts with contract.fluid.yaml — the single source of truth for your data product's entire lifecycle.

fluidVersion: "0.7.1"
kind: DataProduct
id: example.customer_360
name: Customer 360
domain: analytics

metadata:
  layer: Gold
  owner:
    team: data-platform
    email: platform@example.com

# 1. THE LOGIC — How is it built?
builds:
  - id: transform_customer
    pattern: embedded-logic
    engine: sql
    properties:
      sql: |
        SELECT user_id, email, LTV
        FROM raw.users JOIN raw.orders USING (user_id)

# 2. THE INTERFACE — What does it output?
exposes:
  - exposeId: customer_profiles
    kind: table
    binding:
      platform: snowflake              # ← Change to 'gcp' or 'aws' instantly
      format: snowflake_table
      location:
        database: PROD
        schema: GOLD
        table: CUST_360
    contract:
      schema:
        - name: email
          type: string
          sensitivity: pii             # ← Triggers auto-masking/encryption

# 3. THE GOVERNANCE — Who (or what) can access it?
accessPolicy:
  grants:
    - principal: "group:marketing@example.com"
      permissions: ["read"]

agentPolicy:                           # ← Agentic Era Governance
  allowedModels: ["gpt-4", "claude-3"]
  allowedUseCases: ["analysis", "summarization"]

🔌 Providers — Bring Your Own Cloud

Providers are the bridge between your declarative contract and your target execution environment.

Export-only providers for open data standards: odps, odcs, datamesh-manager.

🛠️ Installation

FLUID Forge is modular. Install only what you need.

pip install fluid-forge                # Minimal — CLI + Local/DuckDB provider
pip install "fluid-forge[gcp]"         # + Google Cloud
pip install "fluid-forge[aws]"         # + AWS
pip install "fluid-forge[snowflake]"   # + Snowflake
pip install "fluid-forge[all]"         # Everything

💡 Tip: We recommend pipx for an isolated global install: pipx install "fluid-forge[all]"

💻 CLI Command Reference

FLUID Forge is designed to feel as natural as git or terraform.

Core Lifecycle

fluid init                           # Scaffold a new Data Product contract
fluid validate contract.fluid.yaml   # Validate schema, dependencies, syntax
fluid plan contract.fluid.yaml       # Generate a deterministic execution plan
fluid apply contract.fluid.yaml      # Execute the plan against your target provider
fluid verify contract.fluid.yaml     # Post-deployment data quality & compliance checks

AI & Code Generation

fluid forge                                  # 🤖 Interactive, AI-powered project creation
fluid generate-airflow contract.fluid.yaml   # Compile contract → native Airflow DAG
fluid generate-pipeline contract.fluid.yaml  # Scaffold transformation code

Governance & Compliance

fluid policy-compile contract.fluid.yaml   # Translate policies → native IAM
fluid contract-tests contract.fluid.yaml   # Run assertion suites

Visualization

fluid graph contract.fluid.yaml   # Graphviz DAG of internal lineage
fluid docs contract.fluid.yaml    # Auto-generate documentation from contract

🎓 Templates

Don't start from scratch. fluid init ships with battle-tested enterprise patterns:

fluid init --template customer-360

🤝 Contributing

FLUID Forge is community-driven. We want your ideas, providers, and pull requests!

1. Fork the repo
2. Run make setup for a full dev environment
3. Check out CONTRIBUTING.md for style guides and architecture overview

Please read our Code of Conduct before participating.

To report security vulnerabilities, see SECURITY.md.

License

Apache License 2.0 · Copyright 2024–2026 Agentics Transformation Pty Ltd