buildflow 0.2.1

BuildFlow, is an open source framework for building large scale systems using Python. All you need to do is describe where your input is coming from and where your output should be written, and BuildFlow handles the rest.

⚒️ BuildFlow

Overview

BuildFlow, is an open source framework for building large scale systems using Python. All you need to do is describe where your input is coming from and where your output should be written, and BuildFlow handles the rest. No configuration outside of the code is required.

Key Features (all provided out-of-the-box):

• Automatic resource creation / management (Infrastructure as Code) powered by Pulumi (Infrastructure from Code)
• Automatic parallelism & concurrency powered by Ray
• Dynamic autoscaling: scale up during high traffic / reduce costs during low traffic

Installation

pip install buildflow

Extra Dependencies

Pulumi Installation

BuildFlow uses Pulumi to manage resources used by your BuildFlow Nodes and Processors. To install Pulumi visit: https://www.pulumi.com/docs/install/

Installing Pulumi unlocks:

• allows BuildFlow to manage resource creation and destruction
• full access to Pulumi API / CLI
• fine-grained control over Pulumi Stacks & Resources

Quick Links

• Docs: https://www.buildflow.dev/docs
• Walkthroughs: https://www.buildflow.dev/docs/walkthroughs/realtime-image-classification
• Discord: https://discordapp.com/invite/wz7fjHyrCA
• Contribute: https://www.buildflow.dev/docs/developers/contribute

Code Health Checks

We use black and ruff with pre-commit hooks to perform health checks. To setup these locally:

• Clone the repo
• Install the dev dependencies like `python -m pip install .[dev]
• Check if pre-commit is installed correctly by running pre-commit --version
• Setup pre-commit to run before every commit on staged changes by running pre-commit install
• Pre-commit can also be ran manually as pre-commit run --all-files