GPU Sentinel
A Moonshine Labs tool
Overview
If you're automating training your large models in the cloud, cost control is critial. How many times have you accidentally left an expensive GPU instance running when the underlying job had crashed, costing you money or capacity with no benefit?
GPU Sentinel is a simple tool that will watch your instance and automatically trigger when GPU utilization drops below a certain amount for a period of time. GPU Sentinel can automatically shutdown or reboot the instance, or simply end its own process so you can do an action yourself.
Installation
$ pip install gpu_sentinel
$ gpu_sentinel --help
Usage
The GPU sentinel has two states, IDLE and ARMED.
When you start the program, it will wait for the GPU to be above a certain utilization for a set amount of time. Once this condition is met, the sentinel will be ARMED. This will let you set the sentinel at any point, and it will only trigger once the GPU has been running for a while.
Once ARMED, the sentinel will wait for the GPU utilization to drop below a certain threshold for a set amount of time. Once this condition is met, the kill_action will occur immediately.
Options:
arm_duration: How many seconds of activity to wait before arming the sentinel.
arm_threshold: What level of utilization is considered activity
kill_duration: How many seconds of inactivity to wait before running the kill function.
kill_threshold: What level of utilization is considered inactivity
kill_action: What to do when the kill trigger is hit {end_process,shutdown,reboot}
gpu_devices: Which GPU devices to average (empty for all)
API
If you would prefer to use integrate this package into your own code, we provide a straightforward API to do so.
from gpu_sentinel import Sentinel, get_gpu_usage
def my_callback_fn():
print("Triggered!")
exit()
# Create the sentinel that watches the values.
sentinel = Sentinel(
arm_duration=10,
arm_threshold=0.7,
kill_duration=60,
kill_threshold=0.7,
kill_fn=my_callback_fn,
)
while True:
# This is the averaged GPU usage of the devices.
gpu_usage = get_gpu_usage(device_ids=[0, 1, 2, 3])
# Add the GPU usage to the sentinel's next state.
sentinel.tick(gpu_usage)
# The sentinel operates on ticks, not seconds, so if we want to check every second
# we must do the timer ourselves.
time.sleep(1)
Current Limitations
- To shutdown/reboot the machine, GPU Sentinel requires sudo permissions or sudo-less shutdown.
- Currently only working on Linux, can add Windows support if there's interest.
Metadata
Release files for gpu-sentinel 0.1.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| gpu_sentinel-0.1.4.tar.gz | 5.7 kB | Without provenance | Details for gpu_sentinel-0.1.4.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| gpu_sentinel-0.1.4-py3-none-any.whl | Python 3 | none | any | Details for gpu_sentinel-0.1.4-py3-none-any.whl |
Total release size: 12.8 kB
Release files / gpu_sentinel-0.1.4.tar.gz
| Download URL | gpu_sentinel-0.1.4.tar.gz |
|---|---|
| Size | 5.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c25eb6c75fd538f2b60fa9b95ad77abfa1227da452b51e91b943ea04a7ecc946
|
|
BLAKE2b-256 checksum How to use checksums |
eec4c6bf93bd41397ce7a10ca9e9b18c9f69f355ad1a4b666c1eba09b8ea95b7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.0
|
Release files / gpu_sentinel-0.1.4-py3-none-any.whl
| Download URL | gpu_sentinel-0.1.4-py3-none-any.whl |
|---|---|
| Size | 7.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
615abff10bc5769e506f72d1653753c910b3251001293d9892de84772f6ac5a5
|
|
BLAKE2b-256 checksum How to use checksums |
a810f1bf2b11227ed2a5632ecda66df2495dfbb7c546d73113c87bfff548be08
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.0
|
Security for gpu-sentinel 0.1.4
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.