buildflow is a unified **batch** and **streaming** framework that turns any python function into a scalable data pipeline that can read from our supported IO resources.
Project description
BuildFlow
BuildFlow, is an open source framework that lets you build a data pipeline by simply attaching a decorator to a Python function. All you need to do is describe where your input is coming from and where your output should be written, and BuildFlow handles the rest. No configuration outside of the code is required.
Key Features:
- Unified batch and streaming Processor API
- Production-grade IO connectors for popular cloud services & storage systems
- IO templates for common data pipelines (e.g. file upload notifications)
- Automatic resource creation / management for popular cloud resources
- Schema validation powered by Python dataclasses and type hints
- Automatic parallelism powered by Ray
Quick Links
- Docs: https://www.buildflow.dev/docs/intro
- Walkthroughs: https://www.buildflow.dev/docs/category/walk-throughs
- Discord: https://discordapp.com/invite/wz7fjHyrCA
Quickstart
Install
pip install buildflow
Example Usage
# Import the buildflow package
import buildflow
from buildflow import Flow
# Create the Flow object
flow = Flow()
# Define your input / output
@flow.processor(
source=buildflow.PubSubSource(subscription='my_subscription'),
sink=buildflow.BigQuerySink(table_id='project.dataset.table'),
)
def stream_processor(pubsub_message):
# TODO(developer): Implement processing logic
...
# The output is sent to the sink provider
return pubsub_message
# Start the processor(s)
flow.run().output()
For a more in depth tutorial see our walkthroughs.
Windows Users
Our runtime is built on Ray, where Windows support is currently in beta. See the Ray docs for more info.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for buildflow-0.0.13.dev1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0aac0ac7dde62b670f047cfe9707088ebf8c3c36f2d2810b7d0afe10090b5989 |
|
MD5 | 1114f431c28ff47ff817c278ce17aaed |
|
BLAKE2b-256 | b9046a5323bdff2a346409fd282de30faedb235e0a4a4bb4dcef4160bc6f3bed |