BigQuery client wrapper with clean API
Project description
BigFlow
Documentation
- What is BigFlow?
- Getting started
- Installing Bigflow
- Help me
- BigFlow tutorial
- CLI
- Configuration
- Project setup and build
- Deployment
- Workflow & Job
- Starter
- Technologies
- Logging
- Roadmap
What is BigFlow?
BigFlow is a Python framework for data processing pipelines on GCP.
The main features are:
- Dockerized deployment environment
- Powerful CLI
- Automated build, deployment, versioning and configuration
- Unified project structure
- Support for the major data processing technologies — Dataproc (Apache Spark), Dataflow (Apache Beam) and BigQuery
- Project starter
Getting started
Start from setting up a development environment. Next, go through the BigFlow tutorial.
Installing BigFlow
Prerequisites. Before you start, make sure you have the following software installed:
- Python == 3.7
- Google Cloud SDK
- Docker Engine
You can install the bigflow
package globally but we recommend to
install it locally with venv
, in your project's folder:
python -m venv .bigflow_env
source .bigflow_env/bin/activate
Install the bigflow
PIP package:
pip install bigflow==1.0.dev67
Test it:
bigflow -h
Read more about BigFlow CLI.
Help me
You can ask questions on our gitter channel or stackoverflow.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file bigflow-1.0.dev79.tar.gz
.
File metadata
- Download URL: bigflow-1.0.dev79.tar.gz
- Upload date:
- Size: 62.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.6.0 requests/2.24.0 setuptools/49.6.0 requests-toolbelt/0.9.1 tqdm/4.50.2 CPython/3.7.9
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8b6df505a68df352c2432222129433fd0791b81d923cea3aa65189640bc29ad7 |
|
MD5 | 168691028dc45215113af351a10dbdf3 |
|
BLAKE2b-256 | 5a80ec879d673ca72eab9340adac81522cb9c0633bd3ff2815fef5f6d01b4930 |
File details
Details for the file bigflow-1.0.dev79-py3-none-any.whl
.
File metadata
- Download URL: bigflow-1.0.dev79-py3-none-any.whl
- Upload date:
- Size: 56.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.2.0 pkginfo/1.6.0 requests/2.24.0 setuptools/49.6.0 requests-toolbelt/0.9.1 tqdm/4.50.2 CPython/3.7.9
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | bfe5d41e8bdfc3ea1eadda1789cf95a24cefbd1bab437ea99464807821d7b18e |
|
MD5 | 598996db65f77a7db1a4762cc69d2cf4 |
|
BLAKE2b-256 | 313a99d06e1e435b535ed65ab4f18608d4d9efd3f074a29e41740d437acb24a8 |