Skip to main content

Command-line utility for creating Dataflow projects

Project description

dataflow-cookiecutter Build Status PyPI

Tired of copy-pasting your ad-hoc Dataflow modules? Then you can use this cookiecutter command-line tool to easily generate standardized Dataflow templates! :zap:


You can install dataflow-cookiecutter from PyPI:

pip install dataflow-cookiecutter

In addition, you can also clone this repository and install locally:

git clone
cd dataflow-cookiecutter
python3 install


You can create a Dataflow project by executing the command:

$ dataflow-cookiecutter new

Choose from a variety of our premade templates. See all available templates by running dataflow-cookiecutter ls. For example, you can create a Google Cloud Storage (GCS) to BigQuery (BQ) pipeline via:

$ dataflow-cookiecuter new --template=GCSToBQ

Lastly, our templates are highly-compatible to your trusty, old cookiecutter command-line tool (be sure to use cookiecutter>=1.7.1!):

$ cookiecutter \
   --directory <directory-to-desired-template> 


  • Why are you still wrapping cookiecutter? This started as my learning project to see how cookiecutter's internals work. While building the alpha version, I realized that I can add more functionality to this CLI more than templating, so wrapping Cookiecutter seems to be a good approach.
  • I already have cookiecutter, can I use it with your templates? Yes of course! Look at the Usage section above! However, ensure that your cookecutter version is >=1.7.1 so that you can use the --directory flag!
  • Why are you using Python 3 for Dataflow templates? It's 2020, we shouldn't be supporting legacy Python anymore. Besides, Dataflow now has streaming support in Python 3. See more developments for Beam support in Python 3 in their issue tracker.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dataflow-cookiecutter-1.0.0a1.tar.gz (4.4 kB view hashes)

Uploaded source

Built Distribution

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page