Flowfile Logo
Flowfile

Main Repository: Edwardvaneechoud/Flowfile
Documentation: Website - Core - Worker - Frontend - Technical Architecture

Flowfile is an open-source data platform that combines a visual pipeline builder, a data catalog with Delta Lake storage, scheduling, Kafka ingestion, sandboxed Python execution, and a Polars-compatible Python API — all in a single pip install.

Quick Start

pip install Flowfile
flowfile run ui

This starts the backend services and opens the visual ETL interface in your browser.

What You Get

  • Visual pipeline builder with 30+ nodes for joins, filters, aggregations, fuzzy matching, pivots, and more
  • Data catalog with Delta Lake storage, version history, and lineage tracking
  • Scheduling — interval-based or triggered by catalog table updates
  • Kafka/Redpanda ingestion as a canvas node with automatic schema inference
  • Sandboxed Python execution in isolated Docker containers
  • Code generation — export visual flows as standalone Python/Polars scripts
  • Flow parameters — ${variable} substitution, configurable via UI or CLI
  • Cloud storage — S3, Azure Data Lake Storage, Google Cloud Storage
  • Database connectivity — PostgreSQL, MySQL, SQL Server, Oracle, DuckDB, and more
  • Python API with Polars-like syntax and visual flow graph generation

Python API

import flowfile as ff
from flowfile import col, open_graph_in_editor

df = ff.from_dict({
    "id": [1, 2, 3, 4, 5],
    "category": ["A", "B", "A", "C", "B"],
    "value": [100, 200, 150, 300, 250]
})

result = df.filter(col("value") > 150).with_columns([
    (col("value") * 2).alias("double_value")
])

# Open the pipeline on the visual canvas
open_graph_in_editor(result.flow_graph)

Common Operations

import flowfile as ff
from flowfile import col, when, lit

# Read from various sources
df = ff.read_csv("data.csv")
df_pq = ff.read_parquet("data.parquet")

# Transform
filtered = df.filter(col("value") > 150)
with_status = df.with_columns([
    when(col("value") > 200).then(lit("High")).otherwise(lit("Low")).alias("status")
])

# Aggregate
by_category = df.group_by("category").agg([
    col("value").sum().alias("total"),
    col("value").mean().alias("average")
])

# Join
joined = df.join(other_df, left_on="id", right_on="product_id")

# Visualize any pipeline
ff.open_graph_in_editor(joined.flow_graph)

Code Generation

Export visual flows as standalone Python/Polars scripts:

Code Generation

Package Components

  • Core Service (flowfile_core) — ETL engine, catalog, scheduler, auth
  • Worker Service (flowfile_worker) — CPU-intensive data processing
  • Web UI — Browser-based visual pipeline builder
  • FlowFrame API (flowfile_frame) — Polars-compatible Python library
  • Scheduler (flowfile_scheduler) — Interval and table-trigger scheduling

CLI

flowfile run ui                              # Start web UI
flowfile run core --host 0.0.0.0             # Start core service
flowfile run worker --host 0.0.0.0           # Start worker service
flowfile run flow pipeline.json              # Run a flow
flowfile run flow pipeline.json --param key=value  # Run with parameters

More Options

Resources

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.16.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for edwardvaneechoud from gravatar.com edwardvaneechoud

Credits

Author: Edward van Eechoud

Requires

Python <3.14, >=3.10

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

flowfile-0.14.0.tar.gz (8.7 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

flowfile-0.14.0-py3-none-any.whl (9.2 MB view details)

Uploaded Python 3

File details

Details for the file flowfile-0.14.0.tar.gz.

File metadata

  • Download URL: flowfile-0.14.0.tar.gz
  • Upload date:
  • Size: 8.7 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for flowfile-0.14.0.tar.gz
Algorithm Hash digest
SHA256 a7f61c167b4b585a7c43e77efc46dc130eeab8a59b59e63dac74690be8a17156
MD5 31ef41da493d057703b5205d5d30e6e4
BLAKE2b-256 c2d983d42ae9b4108ebf34955f6f93fc3b53f97bcd9ed6c682ed94d4fa4d5d5a

See more details on using hashes here.

Provenance

The following attestation bundles were made for flowfile-0.14.0.tar.gz:

Publisher: pypi-release.yml on Edwardvaneechoud/Flowfile

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file flowfile-0.14.0-py3-none-any.whl.

File metadata

  • Download URL: flowfile-0.14.0-py3-none-any.whl
  • Upload date:
  • Size: 9.2 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for flowfile-0.14.0-py3-none-any.whl
Algorithm Hash digest
SHA256 9d978fbb4c3c12e23ff180bfd3c871ddd848b0dc63c04b5aaf3c558a7a273e59
MD5 fa581b5c7e7890398a49cb38732d584a
BLAKE2b-256 c5489aeb4f341c95a9027b8dd8fb4b81f91a4217ed84bb466e1d5fa4e1ff24c3

See more details on using hashes here.

Provenance

The following attestation bundles were made for flowfile-0.14.0-py3-none-any.whl:

Publisher: pypi-release.yml on Edwardvaneechoud/Flowfile

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.16.0

2 files

0.15.4

2 files

0.15.3

2 files

0.15.2

2 files

0.15.1

2 files

0.15.0

2 files

0.14.1

2 files

This release

0.14.0 This release

2 files

0.13.4

2 files

0.13.3

2 files

0.13.2

2 files

0.13.1

2 files

0.13.0

2 files

0.12.8

2 files

0.12.7

2 files

0.12.6

2 files

0.12.5

2 files

0.12.4

2 files

0.12.3

2 files

0.12.2

2 files

0.12.1

2 files

0.12.0

2 files

0.11.4

2 files

0.11.3

2 files

0.11.2

2 files

0.11.1

2 files

0.11.0

2 files

0.10.1

2 files

0.10.0

2 files

0.9.4

2 files

0.9.3

2 files

0.9.2

2 files

0.9.1

2 files

0.9.0

2 files

0.8.2

2 files

0.8.1

2 files

0.8.0

2 files

0.7.2

2 files

0.6.3

2 files

0.6.2

2 files

0.6.1

2 files

0.5.6

2 files

0.5.4

2 files

0.5.3

2 files

0.5.2

2 files

0.5.1

2 files

0.5.0

2 files

0.4.1

2 files

0.4.0

2 files

0.3.10

2 files

0.3.9

2 files

0.3.8

2 files

0.3.7

2 files

0.3.6

2 files

0.3.5

2 files

0.3.4.1

2 files

0.3.4

2 files

0.3.3.2

2 files

0.3.3.1

2 files

0.3.3

2 files

0.3.2

2 files

0.3.1.2

2 files

0.3.1.1

2 files

0.3.1

2 files

0.3.0.3

2 files

0.3.0.2

2 files

0.3.0.1

2 files

0.3.0

2 files

0.2.2

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page