BigFlow
Documentation
- What is BigFlow?
- Getting started
- Installing Bigflow
- Help me
- BigFlow tutorial
- CLI
- Configuration
- Project structure and build
- Deployment
- Workflow & Job
- Starter
- Technologies
- Development
Cookbook
What is BigFlow?
BigFlow is a Python framework for data processing pipelines on GCP.
The main features are:
- Dockerized deployment environment
- Powerful CLI
- Automated build, deployment, versioning and configuration
- Unified project structure
- Support for GCP data processing technologies — Dataflow (Apache Beam) and BigQuery
- Project starter
Getting started
Start from installing BigFlow on your local machine. Next, go through the BigFlow tutorial.
Installing BigFlow
Prerequisites. Before you start, make sure you have the following software installed:
You can install the bigflow package globally, but we recommend
installing it locally with venv, in your project's folder:
python -m venv .bigflow_env
source .bigflow_env/bin/activate
Install the bigflow PIP package:
pip install bigflow[bigquery,dataflow]
Test it:
bigflow -h
Read more about BigFlow CLI.
To interact with GCP you need to set a default project and log in:
gcloud config set project <your-gcp-project-id>
gcloud auth application-default login
Finally, check if your Docker is running:
docker info
Help me
You can ask questions on our gitter channel or stackoverflow.
1 maintainer
Metadata
1 maintainer
Release files for bigflow 1.11.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| bigflow-1.11.3.tar.gz | 113.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| bigflow-1.11.3-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 231.8 kB
Release files / bigflow-1.11.3.tar.gz
| Download URL | bigflow-1.11.3.tar.gz |
|---|---|
| Size | 113.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
21f10e2887d21e81d2e1912160e370ad97de3d431abd1578b2301764500f07e6
|
|
BLAKE2b-256 checksum How to use checksums |
68dd17fb8cf5e4e585d6e8c2978dbf0170f56a8e6a0f0d9f78c0c8b0171d2109
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Release files / bigflow-1.11.3-py3-none-any.whl
| Download URL | bigflow-1.11.3-py3-none-any.whl |
|---|---|
| Size | 118.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b82a78b5d13262c10d943f7b43eaf0bb84b1422a0b1897f25b63a3e9789b9446
|
|
BLAKE2b-256 checksum How to use checksums |
6b9b3727d234ebc3d8977659362188199fc4084138345df184542d40a00253c1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
