Skip to main content

Tools for using and

Project description Documentation Status

Tools for using and



pip install rhg_compute_tools


Kubernetes tools

  • easily spin up a preconfigured cluster with get_cluster(), or flavors with get_micro_cluster(), get_standard_cluster(), get_big_cluster(), or get_giant_cluster().
>>> import rhg_compute_tools.kubernetes as rhgk
>>> cluster, client = rhgk.get_cluster()

Google cloud storage utilities

  • Utilities for managing google cloud storage directories in parallel from the command line or via a python API
>>> import rhg_compute_tools.gcs as gcs
>>> gcs.sync_gcs('my_data_dir', 'gs://my-bucket/my_data_dir')



Bug fixes: * raise error on gsutil nonzero status in rhg_compute_tools.gcs.cp (PR #105)


New features: * Adds google storage directory marker utilities and rctools gcs mkdirs command line app


  • Add dask_kwargs to the rhg_compute_tools.xarray functions


  • Add retry_with_timeout to


  • Drop matplotlib.font_manager._rebuild() call in design.__init__ - no longer supported


  • Refactor datasets_from_delayed to speed up


  • Add function


  • Fix tag kwarg in get_cluster


  • Make the gsutil API consistent, so that we have cp, sync and rm, each of which accept the same args and kwargs
  • Swap bumpversion for setuptools_scm to handle versioning
  • Cast coordinates to dict before gathering in rhg_compute_tools.xarray.dataarrays_from_delayed and rhg_compute_tools.xarray.datasets_from_delayed. This avoids a mysterious memory explosion on the local machine. Also add name in the metadata used by those functions so that the name of each dataarray or Variable is preserved.
  • Use dask-gateway when available when creating a cluster in rhg_compute_tools.kubernetes. Add some tests using a local gateway cluster. TODO: More tests.
  • Add tag kwarg to rhg_compute_tools.kuberentes.get_cluster function (PR #87)


  • ?


  • Add remote scheduler deployment (part of dask_kubernetes 0.10)
  • Remove extraneous GCSFUSE_TOKENS env var no longer used in new worker images
  • Set library thread limits based on how many cpus are available for a single dask thread
  • Change formatting of the extra env_items passed to get_cluster to be a list rather than a list of dict-like name/value pairs


  • Add CLI tools . See rctools gcs repdirstruc --help to start
  • Add new function rhg_compute_tools.gcs.replicate_directory_structure_on_gcs to copy directory trees into GCS. Users can authenticate with cred_file or with default google credentials
  • Fixes to docstrings and metadata
  • Add new function rhg_compute_tools.gcs.rm to remove files/directories on GCS using the API
  • Store one additional environment variable when passing cred_path to rhg_compute_tools.kubernetes.get_cluster so that the API will be authenticated in addition to gsutil


  • Deployment fixes


  • Design tools: use RHG & CIL colors & styles
  • Plotting helpers: generate cmaps with consistent colors & norms, and apply a colorbar to geopandas plots with nonlinear norms
  • Autoscaling fix for kubecluster: switch to dask_kubernetes.KubeCluster to allow use of recent bug fixes


  • Add rhg_compute_tools.gcs.cp_gcs and rhg_compute_tools.gcs.sync_gcs utilities


  • need to figure out how to use this rever thing


  • Bug fix again in rhg_compute_tools.kubernetes.get_worker


  • Bug fix in rhg_compute_tools.kubernetes.get_worker


  • Add xarray from delayed methods in rhg_compute_tools.xarray
  • rhg_compute_tools.gcs.cp_to_gcs now calls gsutil in a subprocess instead of operations. This dramatically improves performance when transferring large numbers of small files
  • Additional cluster creation helpers


  • New google compute helpers (see rhg_compute_tools.gcs.cp_to_gcs, rhg_compute_tools.gcs.get_bucket)
  • New cluster creation helper (see rhg_compute_tools.kubernetes.get_worker)
  • Dask helpers (see rhg_compute_tools.utils submodule)


  • First release on PyPI.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

rhg_compute_tools-1.2.1.tar.gz (43.5 kB view hashes)

Uploaded source

Built Distribution

rhg_compute_tools-1.2.1-py2.py3-none-any.whl (29.1 kB view hashes)

Uploaded py2 py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page