Tools for using compute.rhg.com and compute.impactlab.org
Project description
Tools for using compute.rhg.com and compute.impactlab.org
Free software: MIT license
Documentation: https://rhg-compute-tools.readthedocs.io.
Installation
pip:
pip install rhg_compute_tools
Features
Kubernetes tools
easily spin up a preconfigured cluster with get_cluster(), or flavors with get_micro_cluster(), get_standard_cluster(), get_big_cluster(), or get_giant_cluster().
>>> import rhg_compute_tools.kubernetes as rhgk
>>> cluster, client = rhgk.get_cluster()
Google cloud storage utilities
Utilities for managing google cloud storage directories in parallel from the command line or via a python API
>>> import rhg_compute_tools.gcs as gcs
>>> gcs.sync_gcs('my_data_dir', 'gs://my-bucket/my_data_dir')
History
v0.2.0
Add CLI tools . See rctools gcs repdirstruc --help to start
Add new function rhg_compute_tools.gcs.replicate_directory_structure_on_gcs to copy directory trees into GCS. Users can authenticate with cred_file or with default google credentials
Fixes to docstrings and metadata
Add new function rhg_compute_tools.gcs.rm to remove files/directories on GCS using the google.cloud.storage API
Store one additional environment variable when passing cred_path to rhg_compute_tools.kubernetes.get_cluster so that the google.cloud.storage API will be authenticated in addition to gsutil
v0.1.8
Deployment fixes
v0.1.7
Design tools: use RHG & CIL colors & styles
Plotting helpers: generate cmaps with consistent colors & norms, and apply a colorbar to geopandas plots with nonlinear norms
Autoscaling fix for kubecluster: switch to dask_kubernetes.KubeCluster to allow use of recent bug fixes
v0.1.6
Add rhg_compute_tools.gcs.cp_gcs and rhg_compute_tools.gcs.sync_gcs utilities
v0.1.5
need to figure out how to use this rever thing
v0.1.4
Bug fix again in rhg_compute_tools.kubernetes.get_worker
v0.1.3
Bug fix in rhg_compute_tools.kubernetes.get_worker
v0.1.2
Add xarray from delayed methods in rhg_compute_tools.xarray
rhg_compute_tools.gcs.cp_to_gcs now calls gsutil in a subprocess instead of google.storage operations. This dramatically improves performance when transferring large numbers of small files
Additional cluster creation helpers
v0.1.1
New google compute helpers (see rhg_compute_tools.gcs.cp_to_gcs, rhg_compute_tools.gcs.get_bucket)
New cluster creation helper (see rhg_compute_tools.kubernetes.get_worker)
Dask client.map helpers (see rhg_compute_tools.utils submodule)
v0.1.0
First release on PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for rhg_compute_tools-0.2.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7de5cb7f5fc646582e23b7be14feec63e29440c99bc57c744288d036e811adc3 |
|
MD5 | 5d01ead39570289502bf0e1d7c52045d |
|
BLAKE2b-256 | 342cdf4a5b637bdf024ba9e7cf6ef4519a19d96a71b02d96fae8a1efec531efc |