BigFlow
Documentation
- What is BigFlow?
- Getting started
- Installing Bigflow
- Help me
- BigFlow tutorial
- CLI
- Configuration
- Project structure and build
- Deployment
- Workflow & Job
- Starter
- Technologies
- Development
Cookbook
What is BigFlow?
BigFlow is a Python framework for data processing pipelines on GCP.
The main features are:
- Dockerized deployment environment
- Powerful CLI
- Automated build, deployment, versioning and configuration
- Unified project structure
- Support for GCP data processing technologies — Dataflow (Apache Beam) and BigQuery
- Project starter
Getting started
Start from installing BigFlow on your local machine. Next, go through the BigFlow tutorial.
Installing BigFlow
Prerequisites. Before you start, make sure you have the following software installed:
You can install the bigflow package globally, but we recommend
installing it locally with venv, in your project's folder:
python -m venv .bigflow_env
source .bigflow_env/bin/activate
Install the bigflow PIP package:
pip install bigflow[bigquery,dataflow]
Test it:
bigflow -h
Read more about BigFlow CLI.
To interact with GCP you need to set a default project and log in:
gcloud config set project <your-gcp-project-id>
gcloud auth application-default login
Finally, check if your Docker is running:
docker info
Help me
You can ask questions on our gitter channel or stackoverflow.
Metadata
Release files for bigflow 1.11.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| bigflow-1.11.3.tar.gz | 113.2 kB | Without provenance | Details for bigflow-1.11.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| bigflow-1.11.3-py3-none-any.whl | Python 3 | none | any | Details for bigflow-1.11.3-py3-none-any.whl |
Total release size: 231.8 kB
Release files / bigflow-1.11.3.tar.gz
| Download URL | bigflow-1.11.3.tar.gz |
|---|---|
| Size | 113.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
21f10e2887d21e81d2e1912160e370ad97de3d431abd1578b2301764500f07e6
|
|
BLAKE2b-256 checksum How to use checksums |
68dd17fb8cf5e4e585d6e8c2978dbf0170f56a8e6a0f0d9f78c0c8b0171d2109
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Release files / bigflow-1.11.3-py3-none-any.whl
| Download URL | bigflow-1.11.3-py3-none-any.whl |
|---|---|
| Size | 118.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b82a78b5d13262c10d943f7b43eaf0bb84b1422a0b1897f25b63a3e9789b9446
|
|
BLAKE2b-256 checksum How to use checksums |
6b9b3727d234ebc3d8977659362188199fc4084138345df184542d40a00253c1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Security for bigflow 1.11.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.