Skip to main content
Python Software Foundation 20th Year Anniversary Fundraiser  Donate today!

A Python 3 tool to harness the google cloud to run shell scripts

Project description


A Python 3 tool to harness the google cloud to run shell scripts. In particular, if you have

  1. a docker image and
  2. many independent, idempotent tasks for that image written as shell scripts,

Turbine allows you to run those tasks on many VMs in parallel on GCE. See this jupyter notebook for detailed usage.

Turbine is a lot like

  • Google Cloud Run except you don't need to hit an HTTP endpoint and it supports large machine types and accelerators;
  • Google Cloud Functions except it supports large machine types and accelerators; and
  • Batch on GKE except I understand it, it is significantly lighter-weight than Kubernetes, and it provides a path for TPU access.


Turbine consists of two components:

  1. A controller on the local machine to setup cloud components, enter tasks, and provision VMs.
  2. A small shim to run inside of the docker image and retrieve tasks.

For most users, the recommended method to install the controller is via pip:

pip install turbine

To attach turbine to a docker image, simply install turbine on the image via pip:

RUN pip install turbine

Note that the ENTRYPOINT of the docker image will be overwritten when used by the Turbine controller.


  • google-cloud-pubsub
  • google-cloud-storage
  • google-cloud-logging
  • google-api-python-client

See requirements.txt for detailed version information, if needed.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for turbine, version 1.3.4
Filename, size File type Python version Upload date Hashes
Filename, size turbine-1.3.4.tar.gz (11.6 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page