buildflow 0.1.0

buildflow is a unified **batch** and **streaming** framework that turns any python function into a scalable data pipeline that can read from our supported IO resources.

BuildFlow

BuildFlow, is an open source framework that lets you build a data pipeline by simply attaching a decorator to a Python function. All you need to do is describe where your input is coming from and where your output should be written, and BuildFlow handles the rest. No configuration outside of the code is required.

Key Features:

• Unified batch and streaming Processor API
• Production-grade IO connectors for popular cloud services & storage systems
• IO templates for common data pipelines (e.g. file upload notifications)
• Automatic resource creation / management for popular cloud resources
• Schema validation powered by Python dataclasses and type hints
• Automatic parallelism powered by Ray

Quick Links

• Docs: https://www.buildflow.dev/docs
• Walkthroughs: https://www.buildflow.dev/docs/category/walk-throughs
• Discord: https://discordapp.com/invite/wz7fjHyrCA

Quickstart

Install

pip install buildflow

Example Usage

# Import the buildflow package
import buildflow
from buildflow import Flow

# Create the Flow object
flow = Flow()

# Define your input / output
@flow.processor(
   source=buildflow.PubSubSource(subscription='my_subscription'),
   sink=buildflow.BigQuerySink(table_id='project.dataset.table'),
)
def stream_processor(pubsub_message):
  # TODO(developer): Implement processing logic
  ...
  # The output is sent to the sink provider
  return pubsub_message

# Start the processor(s)
flow.run().output()

For a more in depth tutorial see our walkthroughs.

Windows Users

Our runtime is built on Ray, where Windows support is currently in beta. See the Ray docs for more info.

Code Health Checks

We use black and ruff with pre-commit hooks to perform health checks. To setup these locally:

• Clone the repo
• Install the dev dependencies like `python -m pip install .[dev]
• Check if pre-commit is installed correctly by running pre-commit --version
• Setup pre-commit to run before every commit on staged changes by running pre-commit install
• Pre-commit can also be ran manually as pre-commit run --all-files