Skip to main content

Command-line utility for creating Dataflow projects

Project description

dataflow-cookiecutter Build Status PyPI

Tired of copy-pasting your ad-hoc Dataflow modules? Then you can use this cookiecutter command-line tool to easily generate standardized Dataflow templates! :zap:

Installation

You can install dataflow-cookiecutter from PyPI:

pip install dataflow-cookiecutter

In addition, you can also clone this repository and install locally:

git clone https://github.com/ljvmiranda921/dataflow-cookiecutter.git
cd dataflow-cookiecutter
python3 setup.py install

Usage

You can create a Dataflow project by executing the command:

$ dataflow-cookiecutter new

Choose from a variety of our premade templates. See all available templates by running dataflow-cookiecutter ls. For example, you can create a Google Cloud Storage (GCS) to BigQuery (BQ) pipeline via:

$ dataflow-cookiecuter new --template=GCSToBQ

Lastly, our templates are highly-compatible to your trusty, old cookiecutter command-line tool (be sure to use cookiecutter>=1.7.1!):

$ cookiecutter https://github.com/ljvmiranda921/dataflow-cookiecutter \
   --directory <directory-to-desired-template> 

FAQ

  • Why are you still wrapping cookiecutter? This started as my learning project to see how cookiecutter's internals work. While building the alpha version, I realized that I can add more functionality to this CLI more than templating, so wrapping Cookiecutter seems to be a good approach.
  • I already have cookiecutter, can I use it with your templates? Yes of course! Look at the Usage section above! However, ensure that your cookecutter version is >=1.7.1 so that you can use the --directory flag!
  • Why are you using Python 3 for Dataflow templates? It's 2020, we shouldn't be supporting legacy Python anymore. Besides, Dataflow now has streaming support in Python 3. See more developments for Beam support in Python 3 in their issue tracker.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dataflow-cookiecutter-1.0.0a1.tar.gz (4.4 kB view details)

Uploaded Source

Built Distribution

dataflow_cookiecutter-1.0.0a1-py3-none-any.whl (6.5 kB view details)

Uploaded Python 3

File details

Details for the file dataflow-cookiecutter-1.0.0a1.tar.gz.

File metadata

  • Download URL: dataflow-cookiecutter-1.0.0a1.tar.gz
  • Upload date:
  • Size: 4.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.41.0 CPython/3.6.8

File hashes

Hashes for dataflow-cookiecutter-1.0.0a1.tar.gz
Algorithm Hash digest
SHA256 400b4105cfb29c78921116e0d6ed644c067fcb16c17545c94a6d05ea2a7173ab
MD5 a3ad61b0c265169ea4d775ca349570a9
BLAKE2b-256 b850808aeda67e6a7865b6cda0aee96f41bf265842b71c1f605c1a75def7c4fa

See more details on using hashes here.

File details

Details for the file dataflow_cookiecutter-1.0.0a1-py3-none-any.whl.

File metadata

  • Download URL: dataflow_cookiecutter-1.0.0a1-py3-none-any.whl
  • Upload date:
  • Size: 6.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.41.0 CPython/3.6.8

File hashes

Hashes for dataflow_cookiecutter-1.0.0a1-py3-none-any.whl
Algorithm Hash digest
SHA256 cdbcea8098cf103588afd97c4ca89fcda1526dea3da0289583cca27d259daace
MD5 52d6ff8289bc9c6194ce8c19c42a5974
BLAKE2b-256 84ed439c9640cad4d951df8c95cb7c4e282a4458871ad82e1d0823696fdbc309

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page