factl

A CLI tool for deploying and orchestrating data pipelines in Microsoft Fabric using a declarative, Git-backed workflow model. Define pipelines as YAML, validate them locally, and deploy consistently to any environment — no UI clicking required.

What is this?

factl is a declarative data orchestration framework for Microsoft Fabric. Instead of building pipelines by clicking through the Fabric UI, you define processors and workflows in YAML, version them in Git, and factl handles compilation, deployment, and scheduling. Orchestration runs in a common workspace; medallion Lakehouses and Warehouses live in separate data workspaces, connected through parameterized references.

Core model: processors and workflows

factl separates your data platform into two layers:

Processors are reusable building blocks — Notebooks, DataPipelines, or any Fabric item type. You deploy each processor once, then reference it by name in any number of workflows. Per-workflow parameters control what it does with no processor code changes.

Workflows are YAML files that declare which processors run, in what order, on what schedule, and with what parameters. factl compiles the YAML into Fabric-native DataPipeline definitions and deploys them.

Processor items        Workflow YAML
(Notebooks,            (references processors
 DataPipelines,         by name with params
 custom types)          + schedule)

      │                      │
      │         ┌────────────▼──────────┐
      │         │   Framework Compiler  │
      │         └────────────┬──────────┘
      │                      │
      ▼                      ▼
┌──────────────────────────────────────┐
│          Common Workspace            │
│  (processors, workflows, LH_CTL,    │
│   common items)                       │
│  ┌─────────────┐ ┌────────────────┐  │
│  │ processors/ │ │   workflows/   │  │
│  └─────────────┘ └────────────────┘  │
└────────────┬─────────────────────────┘
             │  workspace_id / lakehouse_id
             │  via params + parameter files
             ▼
┌──────────────────────────────────────┐
│          Data Workspace(s)            │
│  (medallion Lakehouses, Warehouses,   │
│   SQL databases — actual data)        │
│  ┌──────┐ ┌──────┐ ┌──────┐          │
│  │ LH_DP│ │ LH_DP│ │ LH_DP│          │
│  │ .brz │ │ .slv │ │ .gld │          │
│  └──────┘ └──────┘ └──────┘          │
└──────────────────────────────────────┘

The problem it solves: Fabric pipelines built through the UI have no single source of truth, drift between environments, and can't be reviewed or versioned. factl moves the orchestration layer — processors, workflows, control assets, common items, and metadata — into Git, where it can be versioned, reviewed, and deployed consistently. The data itself (Lakehouses, Warehouses, SQL databases) lives in separate data workspaces, accessed by processors through parameterized workspace and Lakehouse IDs.

Where it fits: factl sits between your Git repository and your Fabric workspaces. You write YAML, commit to Git, and factl handles the rest — compilation, parameterization, deployment, and schedule management. factl deploys orchestration (processors, workflows, common items) to a common workspace, while your data Lakehouses and Warehouses live in separate data workspaces — referenced by processors through configurable workspace and Lakehouse IDs.

Who it's for: Data engineers and platform teams building ELT/ETL pipelines in Microsoft Fabric who want repeatable, reviewable, and automated deployments.

Why use it?

• No framework to build from scratch. factl provides a complete deployment and orchestration system — workflow compilation, parameter management, environment promotion, and schedule control are all built in.
• Reusable processors with declarative parameters. Processors (Notebooks, DataPipelines, or any Fabric item type with a custom template) are deployed once and composed into workflows by name. Each workflow configures the processor through params — the same processor behaves differently per workflow with no code changes.
• Write once, deploy anywhere. Environment differences (workspace IDs, Lakehouse IDs, connection strings) live in configuration files, not in pipeline code. The same workflow YAML deploys to dev, test, and prod.
• Standardized pipeline development. Every workflow follows the same declarative format. New team members can understand a pipeline by reading its YAML file.
• Personal workspace isolation. Each developer tests changes in their own Fabric workspace using factl self commands. Shared environments are never touched during development.
• Data workspaces separate from orchestration. Medallion Lakehouses, Warehouses, and SQL databases live in dedicated data workspaces — isolated from the common workspace where processors and workflows run. Processors reach data workspaces through parameterized workspace and Lakehouse IDs.
• Fail early, fix locally. Workflow YAML is validated at parse time — dependency cycles, missing references, and config errors are caught before anything touches Fabric.
• Schedules managed declaratively. Enable or disable schedules per environment. Schedule definitions live in the same YAML file as the workflow.

Quick Start

Prerequisites

• Python 3.10 or later
• A Microsoft Fabric capacity with at least one workspace
• Azure CLI authenticated with Fabric access (az login)

1. Install factl

git clone https://github.com/<your-org>/factl.git
cd factl
pip install -e .

2. Initialize repo configuration

factl config init

This creates three files under .config/.factl/:

• project.yaml — repository structure, deployment paths, item types
• targets.yaml — shared environment common workspace IDs (dev, test, prd)
• variables.yaml — per-environment variables available in workflow YAML

Edit these files to match your Fabric setup:

.config/.factl/targets.yaml — add your shared common workspace IDs:

version: 1
personal_parameter_env: dev
targets:
  dev:
    com_workspace_id: <your-dev-workspace-guid>
    force_disable_schedules: false
    meta_database:
      host: <your-sql-endpoint>
      name: DB_META
  test:
    com_workspace_id: <your-test-workspace-guid>
    force_disable_schedules: true
  prd:
    com_workspace_id: <your-prod-workspace-guid>
    force_disable_schedules: false

.config/.factl/project.yaml — update at minimum:

• project.repo_url — your Git repository URL
• deployment.common.parameter_path — path to your fabric-cicd parameter file
• deployment.common.control.lakehouse.name — your control Lakehouse name

3. Create your developer profile

A profile links you to a personal common workspace for development and testing:

factl profile set bs \
  --com-workspace-id <your-personal-workspace-guid> \
  --display-name "Your Name"

This creates ~/.factl/profiles.yaml and sets your profile as active.

4. Deploy common items to your personal workspace

Common items are Fabric artifacts like Notebooks, Environments, and Spark Job Definitions that live under fabric/com/:

factl self deploy com

This publishes all items from fabric/com/ to your personal workspace using the parameter file configured in project.yaml.

5. Deploy control assets and workflows

Control assets (dbt models, configuration files) are uploaded to the control Lakehouse (LH_CTL) in your workspace:

factl self deploy ctl --auto-create

Orchestration compiles your workflow YAML into Fabric DataPipelines and deploys them:

factl self deploy orc

6. Verify in Fabric

Open your personal workspace in the Fabric portal. You should see:

• Fabric items from fabric/com/ deployed to the workspace
• A controls folder with your control assets in the control Lakehouse
• Compiled DataPipelines under the workflows/ folder
• Data Lakehouses (e.g., LH_DP) referenced by processors may live in a separate data workspace, wired via parameter files

Example workflow

Here's a complete workflow YAML (by default authored under controls/workflows/):

workflows:
  - name: "WF_NycTaxiWeatherDaily"
    description: "NYC taxi + NOAA weather medallion processing."
    schedules:
      - enabled: false
        schedule_type: daily
        times:
          - "06:00"
    processors:
      - name: "NB_IngestTlcTripsToBronze"
        alias: "ingest_tlc"
        depends_on: []
        params:
          dataset_months:
            value: '["2024-01","2024-02"]'
            type: "string"
      - name: "NB_IngestNoaaWeatherToBronze"
        alias: "ingest_weather"
        depends_on: []
        params:
          env:
            value: "{{ env_name }}"
            type: "string"
      - name: "NB_RunDbtTaxiWeather"
        alias: "run_dbt"
        depends_on:
          - "ingest_tlc"
          - "ingest_weather"
        params:
          dbt_project_subpath:
            value: "controls/dbt"
            type: "string"

The workflow declares three processors (two ingests that run in parallel, followed by a dbt run), a daily schedule, and Jinja2 templating for environment-specific values.

The processors themselves (NB_IngestTlcTripsToBronze, NB_IngestNoaaWeatherToBronze, NB_RunDbtTaxiWeather) are Fabric items deployed once to processors/. They can be referenced by name in any workflow. For example, NB_IngestNoaaWeatherToBronze could appear in a different workflow with different start_date and end_date params — same processor, different behavior.

How it works

┌──────────────┐     ┌──────────────┐     ┌───────────────────┐
│ Workflow YAML │ ──▶ │  Framework   │ ──▶ │ Fabric DataPipeline│
│ (controls/)   │     │  Compiler    │     │ JSON definitions   │
└──────────────┘     └──────────────┘     └───────────────────┘
                                                  │
                    ┌──────────────┐               │
                    │ fabric-cicd  │ ◀─────────────┘
                    │ publish      │
                    └──────────────┘
                           │
                    ┌──────▼──────┐     ┌──────────────────┐
                    │   Common    │     │  Data Workspace  │
                    │  Workspace  │ ──▶ │ (Lakehouses,     │
                    │ (processors,│     │  Warehouses,     │
                    │  workflows, │     │  SQL databases)  │
                    │  LH_CTL)    │     │                  │
                    └─────────────┘     └──────────────────┘

1. Deploy processors. Fabric items (Notebooks, DataPipelines, or items backed by custom templates) are deployed to the processors/ folder in the common workspace. Each processor is a reusable building block — deployed once, used by many workflows.
2. Author workflows. You write a workflow YAML file in the configured control folder (default: controls/workflows/) that references processors by name with per-workflow parameters, dependencies, and schedules. The same processor can appear in multiple workflows with different params.
3. Compile. factl self deploy orc loads the YAML, validates it with Pydantic, renders Jinja2 template variables, and compiles the workflow into Fabric DataPipeline JSON using built-in activity templates.
4. Publish. The compiled JSON is published to the common workspace via the fabric-cicd SDK. Parameters are environment-aware — your workspace gets the right workspace and Lakehouse IDs (for both the common and data workspaces) without changing the workflow definition.
5. Schedule. Schedule definitions from the YAML are converted to Fabric schedule JSON and deployed alongside the pipeline.

Four deployment types are supported:

Documentation

• Getting Started — full setup walkthrough
• Architecture — how the framework works
• Configuration — all config files and settings
• Workflows — authoring workflow YAML
• Deployment — deploying to environments and CI/CD
• CLI Commands — complete command reference