Skip to main content

Command-line utility for creating Dataflow projects

Project description

dataflow-cookiecutter Build Status

Tired of copy-pasting your ad-hoc Dataflow modules? Then you can use this cookiecutter command-line tool to easily generate standardized Dataflow templates! :zap:

Installation

You can install dataflow-cookiecutter from PyPI:

pip install dataflow-cookiecutter

In addition, you can also clone this repository and install it locally:

git clone https://github.com/ljvmiranda921/dataflow-cookiecutter.git
cd dataflow-cookiecutter
python3 setup.py install

Usage

You can create a Dataflow project by executing the command:

$ dataflow-cookiecutter new

You can also use some of the premade templates. Say, you wish to create a Google Cloud Storage (GCS) to BigQuery (BQ) pipeline:

$ dataflow-cookiecuter new --template=GCSToBQ

In addition, you can use your trusty, old cookiecutter command-line tool to generate templates (only works for cookiecutter>=1.7.1):

cookiecutter https://github.com/ljvmiranda921/dataflow-cookiecutter \
   --directory <directory-to-desired-template> 

FAQ

  • Why are you still wrapping cookiecutter? This started as a learning project and I want to see how cookiecutter's internals work. In addition, I realized I can add more functionality to the CLI more than templating, so wrapping Cookiecutter seems to be a good approach.
  • I already have cookiecutter, can I use it with your templates? Yes of course! Look at the Usage section above! However, ensure that your cookecutter version is >=1.7.1 so that you can use the --directory flag!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dataflow-cookiecutter-1.0.0a0.tar.gz (2.2 kB view hashes)

Uploaded source

Built Distributions

dataflow_cookiecutter-1.0.0a0-py3.6.egg (19.1 kB view hashes)

Uploaded 3 6

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page