Phlo

The Pythonic lakehouse framework. One Python project to define, run, validate, and inspect lakehouse pipelines.

Phlo is the framework and plugin runtime that ties together familiar lakehouse tools — Dagster, dlt, Sling, dbt, Pandera, Iceberg, Delta, Nessie, Trino, MinIO, and more — behind a single CLI and a coherent product surface called Observatory.

Why Phlo

Most lakehouse projects start in Python and quickly spill into YAML, Compose files, orchestration config, catalog setup, quality checks, and a pile of glue scripts and duplicated config. Phlo keeps those pieces in one project.

Use the phlo CLI to create a project, start the local stack, materialise assets, run quality checks, follow logs, and inspect what happened. Add provider packages when you need them: Dagster for orchestration, dlt or Sling for ingestion, dbt for transforms, Iceberg or Delta for tables, Trino for query, and Observatory for a UI to inspect assets, tables, lineage, quality, services, and logs.

What a Phlo asset looks like

A Phlo asset is ordinary Python with lakehouse metadata attached:

from pathlib import Path

import dlt
import pandas as pd
import phlo

from workflows.schemas.csv import EventsSchema


@phlo.ingest.dlt(
    table_name="events",
    unique_key="event_id",
    validation_schema=EventsSchema,
    group="csv",
    freshness_hours=(1, 24),
)
def csv_events(partition_date: str) -> object:
    events = pd.read_csv(Path("data/events.csv"))
    events["event_id"] = events["id"].astype(str) + "-" + partition_date
    rows = events.to_dict(orient="records")
    return dlt.resource(rows, name="events")

This single function registers a partitioned ingestion asset, validates rows with Pandera, materialises through the configured orchestrator, lands the table in your configured storage and catalog, and becomes visible in Observatory and the catalog CLI; the starter wires the providers you install and generates the local runtime configuration.

Quick Start

Prerequisites

• Python 3.12 or newer; Python 3.12 is tested in CI
• uv
• Docker with Compose v2 (the currently supported path); Podman with a Compose provider is documented as experimental and is not CI-supported

# Create an isolated environment for the quickstart
mkdir phlo-quickstart && cd phlo-quickstart
uv venv
source .venv/bin/activate

# Install Phlo with the default local stack providers
uv pip install "phlo[defaults]"

# Create a project from the CSV batch starter
phlo init my-lakehouse --template csv-batch
cd my-lakehouse
uv pip install -e .

# Generate and start the local lakehouse stack
phlo services init
phlo services start

# Check that services are healthy
phlo services status
phlo doctor --verbose

# Materialize a completed daily partition
phlo materialize dlt_events --partition 2025-01-15

# Verify the table landed in the catalog
phlo catalog tables

# Open the Trino shell on the Iceberg catalog
phlo trino --catalog iceberg

# Stop the local stack when finished
phlo services stop

Capabilities

• Project layout for phlo.yaml, workflows, schemas, transforms, tests, local runtime state, and project plugins.
• Starters for CSV ingestion, REST API ingestion, dbt medallion projects, Sling replication, and Observatory demos.
• Python decorators for registering ingestion, quality, and transformation assets without hand-writing provider boilerplate.
• Local service commands for generating, starting, checking, logging, and stopping the stack.
• Provider packages for Dagster, MinIO, Nessie, Trino, Iceberg, dbt, PostgreSQL, Observatory, and the rest of a working lakehouse.
• Plugin hooks for custom commands, services, assets, resources, catalogs, and Observatory extensions.

How Phlo fits together

Phlo's core stays small. Installed provider packages contribute capabilities through Python entry points; the CLI discovers them in the current project and wires the runtime accordingly.

Documentation

• Installation guide
• First pipeline
• Concepts
• Choose your stack
• Ingest data
• Write a plugin
• Run in production
• CLI reference

Project status

Phlo is alpha. The local development workflow is usable and exercised in CI, but APIs, provider contracts, and the on-disk project layout may change before 1.0. Pin exact versions in production.

The machine-readable v1 support boundary defines the blessed target profile, preview and development-only packages, current runtime evidence, production prerequisites, exclusions, and release-gate status. A v1-target component is not a production-ready guarantee until those gates pass.

Deployment support boundary

The v1 support boundary covers a single-project, single-tenant deployment. Kubernetes-native operation, high availability, and multi-region deployment are outside this support boundary.

Development

Install Python 3.12, uv, and Node.js 24 or later, then run:

make setup
make check

make setup creates the Python environment and installs the locked Observatory and Phlo GitHub writer dependencies. See CONTRIBUTING.md for the full contribution workflow.

Useful local service commands:

phlo services init
phlo services start
phlo services status
phlo services logs -f
phlo services stop
phlo doctor --verbose

Contributing

Issues and pull requests are welcome. See CONTRIBUTING.md for the contribution terms. Run make check locally before opening a PR, and please open an issue first for larger changes so the design can be discussed up front.

Licence

Phlo is open-source software licensed under the GNU Affero General Public License, version 3 or later (AGPL-3.0-or-later). See LICENSE.

If you wish to use Phlo in a proprietary product or service without complying with the AGPL, please contact us about commercial licensing.