Command-line utility for creating Dataflow projects
Project description
dataflow-cookiecutter
Tired of copy-pasting your ad-hoc Dataflow modules? Then you can use this cookiecutter command-line tool to easily generate standardized Dataflow templates! :zap:
Installation
You can install dataflow-cookiecutter
from PyPI:
pip install dataflow-cookiecutter
In addition, you can also clone this repository and install locally:
git clone https://github.com/ljvmiranda921/dataflow-cookiecutter.git
cd dataflow-cookiecutter
python3 setup.py install
Usage
You can create a Dataflow project by executing the command:
$ dataflow-cookiecutter new
Choose from a variety of our premade templates. See all available templates
by running dataflow-cookiecutter ls
. For example, you can create a Google
Cloud Storage (GCS) to BigQuery (BQ) pipeline via:
$ dataflow-cookiecuter new -t GCSToBQ
Lastly, our templates are highly-compatible to your trusty, old
cookiecutter
command-line tool (be sure to use cookiecutter
>=1.7.1!):
$ cookiecutter https://github.com/ljvmiranda921/dataflow-cookiecutter \
--directory <directory-to-desired-template>
FAQ
- Why are you still wrapping cookiecutter? This started as my learning project to see how cookiecutter's internals work. While building the alpha version, I realized that I can add more functionality to this CLI more than templating, so wrapping Cookiecutter seems to be a good approach.
- I already have cookiecutter, can I use it with your templates? Yes of
course! Look at the Usage section above! However, ensure that your
cookecutter version is
>=1.7.1
so that you can use the--directory
flag! - Why are you using Python 3 for Dataflow templates? It's 2020, we shouldn't be supporting legacy Python anymore. Besides, Dataflow now has streaming support in Python 3. See more developments for Beam support in Python 3 in their issue tracker.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dataflow-cookiecutter-1.0.0a3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6bb969501cb0b7d64b866683de7e1134c2ffef52869c0e174b2c21934bf2973d |
|
MD5 | e9d02ddb3c04f944fdcc2eef3add7843 |
|
BLAKE2b-256 | e1d5b7a8beae6f9cee229f1b60c4a32bc1eb6b71c6eec760d810cc3840637abe |
Hashes for dataflow_cookiecutter-1.0.0a3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 17b8f220e3e9188819091fc2f77797d63e9550cdc745b8992635d369bdc3327c |
|
MD5 | 796f1d49758bc2c3f4f21005f7ec3254 |
|
BLAKE2b-256 | 4766feb7a84d5275afe5dba92825c10878abe27e51fa029da582b6b23a635ced |