Command-line utility for creating Dataflow projects
Project description
dataflow-cookiecutter
Tired of copy-pasting your ad-hoc Dataflow modules? Then you can use this cookiecutter command-line tool to easily generate standardized Dataflow templates! :zap:
Installation
You can install dataflow-cookiecutter
from PyPI:
pip install dataflow-cookiecutter
In addition, you can also clone this repository and install locally:
git clone https://github.com/ljvmiranda921/dataflow-cookiecutter.git
cd dataflow-cookiecutter
python3 setup.py install
Usage
You can create a Dataflow project by executing the command:
$ dataflow-cookiecutter new
Choose from a variety of our premade templates. See all available templates
by running dataflow-cookiecutter ls
. For example, you can create a Google
Cloud Storage (GCS) to BigQuery (BQ) pipeline via:
$ dataflow-cookiecuter new --template=GCSToBQ
Lastly, our templates are highly-compatible to your trusty, old
cookiecutter
command-line tool (be sure to use cookiecutter
>=1.7.1!):
$ cookiecutter https://github.com/ljvmiranda921/dataflow-cookiecutter \
--directory <directory-to-desired-template>
FAQ
- Why are you still wrapping cookiecutter? This started as my learning project to see how cookiecutter's internals work. While building the alpha version, I realized that I can add more functionality to this CLI more than templating, so wrapping Cookiecutter seems to be a good approach.
- I already have cookiecutter, can I use it with your templates? Yes of
course! Look at the Usage section above! However, ensure that your
cookecutter version is
>=1.7.1
so that you can use the--directory
flag! - Why are you using Python 3 for Dataflow templates? It's 2020, we shouldn't be supporting legacy Python anymore. Besides, Dataflow now has streaming support in Python 3. See more developments for Beam support in Python 3 in their issue tracker.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dataflow-cookiecutter-1.0.0a2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 22bb0159e7e798b906d0787b64721747b5bbea329f472d77c5dc4ce3b80b583b |
|
MD5 | 2bdff463244d773e0edddf390b7629a2 |
|
BLAKE2b-256 | efa26848f2d7c99157934acb6c39c23bbbcdf9cab01668053313c16bab547271 |
Hashes for dataflow_cookiecutter-1.0.0a2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9ae56fee78bd6e0e23ba713e2fe81d8cd55c7d85339c608449eb651e4a5d6c03 |
|
MD5 | a87205cc4f4eb3c6608b3c544b419816 |
|
BLAKE2b-256 | 4d290908545025575dab3029516959baa83571e2add2ff7e54870217692de6aa |