Flowfile

Main Repository: Edwardvaneechoud/Flowfile
Documentation: Website - Core - Worker - Frontend - Technical Architecture

Flowfile is an open-source data platform that combines a visual pipeline builder, a data catalog with Delta Lake storage, scheduling, Kafka ingestion, sandboxed Python execution, and a Polars-compatible Python API — all in a single pip install.

Quick Start

pip install Flowfile
flowfile run ui

This starts the backend services and opens the visual ETL interface in your browser.

What You Get

Visual pipeline builder with 30+ nodes for joins, filters, aggregations, fuzzy matching, pivots, and more
Data catalog with Delta Lake storage, version history, and lineage tracking
Scheduling — interval-based or triggered by catalog table updates
Kafka/Redpanda ingestion as a canvas node with automatic schema inference
Sandboxed Python execution in isolated Docker containers
Code generation — export visual flows as standalone Python/Polars scripts
Flow parameters — ${variable} substitution, configurable via UI or CLI
Cloud storage — S3, Azure Data Lake Storage, Google Cloud Storage
Database connectivity — PostgreSQL, MySQL, SQL Server, Oracle, DuckDB, and more
Python API with Polars-like syntax and visual flow graph generation

Python API

import flowfile as ff
from flowfile import col, open_graph_in_editor

df = ff.from_dict({
    "id": [1, 2, 3, 4, 5],
    "category": ["A", "B", "A", "C", "B"],
    "value": [100, 200, 150, 300, 250]
})

result = df.filter(col("value") > 150).with_columns([
    (col("value") * 2).alias("double_value")
])

# Open the pipeline on the visual canvas
open_graph_in_editor(result.flow_graph)

Common Operations

import flowfile as ff
from flowfile import col, when, lit

# Read from various sources
df = ff.read_csv("data.csv")
df_pq = ff.read_parquet("data.parquet")

# Transform
filtered = df.filter(col("value") > 150)
with_status = df.with_columns([
    when(col("value") > 200).then(lit("High")).otherwise(lit("Low")).alias("status")
])

# Aggregate
by_category = df.group_by("category").agg([
    col("value").sum().alias("total"),
    col("value").mean().alias("average")
])

# Join
joined = df.join(other_df, left_on="id", right_on="product_id")

# Visualize any pipeline
ff.open_graph_in_editor(joined.flow_graph)

Code Generation

Export visual flows as standalone Python/Polars scripts:

Code Generation

Package Components

Core Service (flowfile_core) — ETL engine, catalog, scheduler, auth
Worker Service (flowfile_worker) — CPU-intensive data processing
Web UI — Browser-based visual pipeline builder
FlowFrame API (flowfile_frame) — Polars-compatible Python library
Scheduler (flowfile_scheduler) — Interval and table-trigger scheduling

CLI

flowfile run ui                              # Start web UI
flowfile run core --host 0.0.0.0             # Start core service
flowfile run worker --host 0.0.0.0           # Start worker service
flowfile run flow pipeline.json              # Run a flow
flowfile run flow pipeline.json --param key=value  # Run with parameters

More Options

Desktop App: Download from GitHub Releases
Docker: docker compose up -d for self-hosted deployments
Browser Demo: demo.flowfile.org (WASM, no server)

Resources

Documentation: Comprehensive guides
Main Repository: Latest code and examples
Technical Architecture: Design overview

Release files for Flowfile 0.18.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for Flowfile 0.18.0
File	Size	Uploaded
flowfile-0.18.0.tar.gz	9.4 MB	Sep 18, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for Flowfile 0.18.0
File	Interpreter	ABI	Platform	Reset
flowfile-0.18.0-py3-none-any.whl 10.0 MB Sep 18, 2026	Python 3	none	any	Details

Total release size: 19.5 MB

Release files / flowfile-0.18.0.tar.gz

Download URL	flowfile-0.18.0.tar.gz
Size	9.4 MB
Tags	Source
SHA-256 checksum How to use checksums	`19d32a46e21e5b67433b34da2b9271738ac1b0ec8309a29395eb96b5f6c6781a`
BLAKE2b-256 checksum How to use checksums	`c92fe00f4210d28368abf2057ed2f8144c0c0360e2bd69fce4c56b16f0bd76a6`
Upload date	Sep 18, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/Edwardvaneechoud/Flowfile
Publishing commit: github.com/Edwardvaneechoud/Flowfile/tree/81ffe50cbb5ef6cd23637addea2aab0ae3929806

Workflow

Publishing configuration: github.com/Edwardvaneechoud/Flowfile/blob/81ffe50cbb5ef6cd23637addea2aab0ae3929806/.github/workflows/pypi-release.yml
Publishing logs: github.com/Edwardvaneechoud/Flowfile/actions/runs/35339720041/attempts/1

Artifact

Subject: flowfile-0.18.0.tar.gz
SHA-256 checksum: 19d32a46e21e5b67433b34da2b9271738ac1b0ec8309a29395eb96b5f6c6781a Verifying attestations

Release files / flowfile-0.18.0-py3-none-any.whl

Download URL	flowfile-0.18.0-py3-none-any.whl
Size	10.0 MB
Tags	Python 3
SHA-256 checksum How to use checksums	`c1093be5349ba6781fefe1e597a30548bbaea64204439a7e309361169849b1e6`
BLAKE2b-256 checksum How to use checksums	`ffc9de47e4a525473148c91a8a3bd5cff4cc72f04f51dd66cb064f41eb3215c9`
Upload date	Sep 18, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/7.0.0 CPython/3.13.14`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/Edwardvaneechoud/Flowfile
Publishing commit: github.com/Edwardvaneechoud/Flowfile/tree/81ffe50cbb5ef6cd23637addea2aab0ae3929806

Workflow

Publishing configuration: github.com/Edwardvaneechoud/Flowfile/blob/81ffe50cbb5ef6cd23637addea2aab0ae3929806/.github/workflows/pypi-release.yml
Publishing logs: github.com/Edwardvaneechoud/Flowfile/actions/runs/35339720041/attempts/1

Artifact

Subject: flowfile-0.18.0-py3-none-any.whl
SHA-256 checksum: c1093be5349ba6781fefe1e597a30548bbaea64204439a7e309361169849b1e6 Verifying attestations