Tools for using compute.rhg.com and compute.impactlab.org
Project description
Tools for using compute.rhg.com and compute.impactlab.org
Free software: MIT license
Documentation: https://rhg-compute-tools.readthedocs.io.
Installation
pip:
pip install rhg_compute_tools
Features
Kubernetes tools
easily spin up a preconfigured cluster with get_cluster(), or flavors with get_micro_cluster(), get_standard_cluster(), get_big_cluster(), or get_giant_cluster().
>>> import rhg_compute_tools.kubernetes as rhgk
>>> cluster, client = rhgk.get_cluster()
Google cloud storage utilities
Utilities for managing google cloud storage directories in parallel from the command line or via a python API
>>> import rhg_compute_tools.gcs as gcs
>>> gcs.sync_gcs('my_data_dir', 'gs://my-bucket/my_data_dir')
History
v0.2.1
Add remote scheduler deployment (part of dask_kubernetes 0.10)
Remove extraneous GCSFUSE_TOKENS env var no longer used in new worker images
Set library thread limits based on how many cpus are available for a single dask thread
Change formatting of the extra env_items passed to get_cluster to be a list rather than a list of dict-like name/value pairs
v0.2.0
Add CLI tools . See rctools gcs repdirstruc --help to start
Add new function rhg_compute_tools.gcs.replicate_directory_structure_on_gcs to copy directory trees into GCS. Users can authenticate with cred_file or with default google credentials
Fixes to docstrings and metadata
Add new function rhg_compute_tools.gcs.rm to remove files/directories on GCS using the google.cloud.storage API
Store one additional environment variable when passing cred_path to rhg_compute_tools.kubernetes.get_cluster so that the google.cloud.storage API will be authenticated in addition to gsutil
v0.1.8
Deployment fixes
v0.1.7
Design tools: use RHG & CIL colors & styles
Plotting helpers: generate cmaps with consistent colors & norms, and apply a colorbar to geopandas plots with nonlinear norms
Autoscaling fix for kubecluster: switch to dask_kubernetes.KubeCluster to allow use of recent bug fixes
v0.1.6
Add rhg_compute_tools.gcs.cp_gcs and rhg_compute_tools.gcs.sync_gcs utilities
v0.1.5
need to figure out how to use this rever thing
v0.1.4
Bug fix again in rhg_compute_tools.kubernetes.get_worker
v0.1.3
Bug fix in rhg_compute_tools.kubernetes.get_worker
v0.1.2
Add xarray from delayed methods in rhg_compute_tools.xarray
rhg_compute_tools.gcs.cp_to_gcs now calls gsutil in a subprocess instead of google.storage operations. This dramatically improves performance when transferring large numbers of small files
Additional cluster creation helpers
v0.1.1
New google compute helpers (see rhg_compute_tools.gcs.cp_to_gcs, rhg_compute_tools.gcs.get_bucket)
New cluster creation helper (see rhg_compute_tools.kubernetes.get_worker)
Dask client.map helpers (see rhg_compute_tools.utils submodule)
v0.1.0
First release on PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for rhg_compute_tools-0.2.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b1a6bf8732696d6fa9115e3a53dc18631d3ed31a77c5e4bbde10edc70757afeb |
|
MD5 | 8ad5009e5a0685cd821c094d885d63dc |
|
BLAKE2b-256 | aa64eabc16994e00fc78706800a8ac9663c962a3a9bbb21e49e794f0030ccc5c |