phlo 0.11.0

Lakehouse platform

Phlo

The Pythonic lakehouse framework. One Python project to define, run, validate, and inspect lakehouse pipelines.

Phlo is the framework and plugin runtime that ties together familiar lakehouse tools — Dagster, dlt, Sling, dbt, Pandera, Iceberg, Delta, Nessie, Trino, MinIO, and more — behind a single CLI and a coherent product surface called Observatory.

---

Why Phlo

Most lakehouse projects start in Python and quickly spill into YAML, Compose files, orchestration config, catalog setup, quality checks, and a pile of glue scripts and duplicated config. Phlo keeps those pieces in one project.

Use the phlo CLI to create a project, start the local stack, materialize assets, run quality checks, follow logs, and inspect what happened. Add provider packages when you need them: Dagster for orchestration, dlt or Sling for ingestion, dbt for transforms, Iceberg or Delta for tables, Trino for query, and Observatory for a UI to inspect assets, tables, lineage, quality, services, and logs.

What a Phlo asset looks like

A Phlo asset is ordinary Python with lakehouse metadata attached:

from pathlib import Path

import dlt
import pandas as pd
import phlo

from workflows.schemas.csv import EventsSchema


@phlo.ingestion(
    table_name="events",
    unique_key="event_id",
    validation_schema=EventsSchema,
    group="csv",
    freshness_hours=(1, 24),
)
def csv_events(partition_date: str) -> object:
    events = pd.read_csv(Path("data/events.csv"))
    events["event_id"] = events["id"].astype(str) + "-" + partition_date
    rows = events.to_dict(orient="records")
    return dlt.resource(rows, name="events")

This single function registers a partitioned ingestion asset, validates rows with Pandera, materializes through the configured orchestrator, lands the table in your configured storage and catalog, and becomes visible in Observatory and the catalog CLI — no separate orchestration, schema, Compose, or catalog wiring needed.

Quick Start

Prerequisites

• Python 3.11 or later
• uv
• Docker with Compose v2, or Podman with a Compose provider

# Create an isolated environment for the quickstart
mkdir phlo-quickstart && cd phlo-quickstart
uv venv
source .venv/bin/activate

# Install Phlo with the default local stack providers
uv pip install "phlo[defaults]"

# Create a project from the CSV batch starter
phlo init my-lakehouse --template csv-batch
cd my-lakehouse
uv pip install -e .

# Generate and start the local lakehouse stack
phlo services init
phlo services start

# Check that services are healthy
phlo services status
phlo doctor --verbose

# Materialize a completed daily partition
phlo materialize dlt_events --partition 2025-01-15

# Verify the table landed in the catalog
phlo catalog tables

# Stop the local stack when finished
phlo services stop

Capabilities

• Project layout for phlo.yaml, workflows, schemas, transforms, tests, local runtime state, and project plugins.
• Starters for CSV ingestion, REST API ingestion, dbt medallion projects, Sling replication, and Observatory demos.
• Python decorators for registering ingestion, quality, and transformation assets without hand-writing provider boilerplate.
• Local service commands for generating, starting, checking, logging, and stopping the stack.
• Provider packages for Dagster, MinIO, Nessie, Trino, Iceberg, dbt, PostgreSQL, Observatory, and the rest of a working lakehouse.
• Plugin hooks for custom commands, services, assets, resources, catalogs, and Observatory extensions.

How Phlo fits together

Phlo's core stays small. Installed provider packages contribute capabilities through Python entry points; the CLI discovers them in the current project and wires the runtime accordingly.

Area	Intent	Provider examples
Pipeline authoring	Define ingestion assets, schemas, checks, and transforms	phlo-dlt, phlo-sling, phlo-pandera, phlo-dbt
Runtime services	Start the local lakehouse stack without hand-written Compose files	phlo-dagster, phlo-postgres, phlo-minio, phlo-nessie, phlo-trino
Table & catalog layer	Store, version, and query lakehouse tables	phlo-iceberg, phlo-delta, phlo-clickhouse, phlo-openmetadata
Product surfaces	Inspect and control assets, tables, lineage, quality, services, and logs	phlo-api, phlo-observatory, phlo-mcp
Serving & BI	Expose lakehouse data to apps and analysts	phlo-hasura, phlo-postgrest, phlo-pgweb, phlo-superset
Observability	Export telemetry, logs, metrics, and alerts	phlo-otel, phlo-prometheus, phlo-loki, phlo-grafana, phlo-alerting
Development	Test and validate projects and provider integrations	phlo-testing

Documentation

• Installation Guide
• Quickstart Guide
• Core Concepts
• Choosing Components
• Workflow Development
• Plugin Development
• Operations Guide
• CLI Reference

Project status

Phlo is alpha. The local development workflow is usable and exercised in CI, but APIs, provider contracts, and the on-disk project layout may change before 1.0. Pin exact versions in production.

Development

uv pip install -e .
make check

Useful local service commands:

phlo services init
phlo services start
phlo services status
phlo services logs -f
phlo services stop
phlo doctor --verbose

Contributing

Issues and pull requests are welcome. Run make check locally before opening a PR, and please open an issue first for larger changes so the design can be discussed up front.