Skip to main content

Tools for using compute.rhg.com and compute.impactlab.org

Project description

https://img.shields.io/pypi/v/rhg_compute_tools.svg https://img.shields.io/travis/RhodiumGroup/rhg_compute_tools.svg Documentation Status

Tools for using compute.rhg.com and compute.impactlab.org

Installation

pip:

pip install rhg_compute_tools

Features

Kubernetes tools

  • easily spin up a preconfigured cluster with get_cluster(), or flavors with get_micro_cluster(), get_standard_cluster(), get_big_cluster(), or get_giant_cluster().
>>> import rhg_compute_tools.kubernetes as rhgk
>>> cluster, client = rhgk.get_cluster()

Google cloud storage utilities

  • Utilities for managing google cloud storage directories in parallel from the command line or via a python API
>>> import rhg_compute_tools.gcs as gcs
>>> gcs.sync_gcs('my_data_dir', 'gs://my-bucket/my_data_dir')

History

v0.2.1

  • Add remote scheduler deployment (part of dask_kubernetes 0.10)
  • Remove extraneous GCSFUSE_TOKENS env var no longer used in new worker images
  • Set library thread limits based on how many cpus are available for a single dask thread
  • Change formatting of the extra env_items passed to get_cluster to be a list rather than a list of dict-like name/value pairs

v0.2.0

  • Add CLI tools . See rctools gcs repdirstruc --help to start
  • Add new function rhg_compute_tools.gcs.replicate_directory_structure_on_gcs to copy directory trees into GCS. Users can authenticate with cred_file or with default google credentials
  • Fixes to docstrings and metadata
  • Add new function rhg_compute_tools.gcs.rm to remove files/directories on GCS using the google.cloud.storage API
  • Store one additional environment variable when passing cred_path to rhg_compute_tools.kubernetes.get_cluster so that the google.cloud.storage API will be authenticated in addition to gsutil

v0.1.8

  • Deployment fixes

v0.1.7

  • Design tools: use RHG & CIL colors & styles
  • Plotting helpers: generate cmaps with consistent colors & norms, and apply a colorbar to geopandas plots with nonlinear norms
  • Autoscaling fix for kubecluster: switch to dask_kubernetes.KubeCluster to allow use of recent bug fixes

v0.1.6

  • Add rhg_compute_tools.gcs.cp_gcs and rhg_compute_tools.gcs.sync_gcs utilities

v0.1.5

  • need to figure out how to use this rever thing

v0.1.4

  • Bug fix again in rhg_compute_tools.kubernetes.get_worker

v0.1.3

  • Bug fix in rhg_compute_tools.kubernetes.get_worker

v0.1.2

  • Add xarray from delayed methods in rhg_compute_tools.xarray
  • rhg_compute_tools.gcs.cp_to_gcs now calls gsutil in a subprocess instead of google.storage operations. This dramatically improves performance when transferring large numbers of small files
  • Additional cluster creation helpers

v0.1.1

  • New google compute helpers (see rhg_compute_tools.gcs.cp_to_gcs, rhg_compute_tools.gcs.get_bucket)
  • New cluster creation helper (see rhg_compute_tools.kubernetes.get_worker)
  • Dask client.map helpers (see rhg_compute_tools.utils submodule)

v0.1.0

  • First release on PyPI.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for rhg-compute-tools, version 0.2.1
Filename, size File type Python version Upload date Hashes
Filename, size rhg_compute_tools-0.2.1-py2.py3-none-any.whl (17.5 kB) File type Wheel Python version py2.py3 Upload date Hashes View
Filename, size rhg_compute_tools-0.2.1.tar.gz (24.9 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page