Skip to main content

GPU Sentinel

A Moonshine Labs tool

Overview

If you're automating training your large models in the cloud, cost control is critial. How many times have you accidentally left an expensive GPU instance running when the underlying job had crashed, costing you money or capacity with no benefit?

GPU Sentinel is a simple tool that will watch your instance and automatically trigger when GPU utilization drops below a certain amount for a period of time. GPU Sentinel can automatically shutdown or reboot the instance, or simply end its own process so you can do an action yourself.

Installation

$ pip install gpu_sentinel
$ gpu_sentinel --help

Usage

The GPU sentinel has two states, IDLE and ARMED.

When you start the program, it will wait for the GPU to be above a certain utilization for a set amount of time. Once this condition is met, the sentinel will be ARMED. This will let you set the sentinel at any point, and it will only trigger once the GPU has been running for a while.

Once ARMED, the sentinel will wait for the GPU utilization to drop below a certain threshold for a set amount of time. Once this condition is met, the kill_action will occur immediately.

Options:

arm_duration: How many seconds of activity to wait before arming the sentinel.
arm_threshold: What level of utilization is considered activity
kill_duration: How many seconds of inactivity to wait before running the kill function.
kill_threshold: What level of utilization is considered inactivity
kill_action: What to do when the kill trigger is hit {end_process,shutdown,reboot}
gpu_devices: Which GPU devices to average (empty for all)

API

If you would prefer to use integrate this package into your own code, we provide a straightforward API to do so.

from gpu_sentinel import Sentinel, get_gpu_usage

def my_callback_fn():
    print("Triggered!")
    exit()

# Create the sentinel that watches the values.
sentinel = Sentinel(
    arm_duration=10,
    arm_threshold=0.7,
    kill_duration=60,
    kill_threshold=0.7,
    kill_fn=my_callback_fn,
)

while True:
    # This is the averaged GPU usage of the devices.
    gpu_usage = get_gpu_usage(device_ids=[0, 1, 2, 3])
    # Add the GPU usage to the sentinel's next state.
    sentinel.tick(gpu_usage)
    # The sentinel operates on ticks, not seconds, so if we want to check every second
    # we must do the timer ourselves.
    time.sleep(1)

Current Limitations

  • To shutdown/reboot the machine, GPU Sentinel requires sudo permissions or sudo-less shutdown.
  • Currently only working on Linux, can add Windows support if there's interest.

Metadata

Release files for gpu-sentinel 0.1.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for gpu-sentinel 0.1.4
File Size Uploaded Provenance Actions
gpu_sentinel-0.1.4.tar.gz 5.7 kB Without provenance Details for gpu_sentinel-0.1.4.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for gpu-sentinel 0.1.4
File Interpreter ABI Platform Actions
gpu_sentinel-0.1.4-py3-none-any.whl Python 3 none any Details for gpu_sentinel-0.1.4-py3-none-any.whl

Total release size: 12.8 kB

Release files / gpu_sentinel-0.1.4.tar.gz

Download URL gpu_sentinel-0.1.4.tar.gz
Size 5.7 kB
Tags Source
SHA-256 checksum
How to use checksums
c25eb6c75fd538f2b60fa9b95ad77abfa1227da452b51e91b943ea04a7ecc946
BLAKE2b-256 checksum
How to use checksums
eec4c6bf93bd41397ce7a10ca9e9b18c9f69f355ad1a4b666c1eba09b8ea95b7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.11.0

Release files / gpu_sentinel-0.1.4-py3-none-any.whl

Download URL gpu_sentinel-0.1.4-py3-none-any.whl
Size 7.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
615abff10bc5769e506f72d1653753c910b3251001293d9892de84772f6ac5a5
BLAKE2b-256 checksum
How to use checksums
a810f1bf2b11227ed2a5632ecda66df2495dfbb7c546d73113c87bfff548be08
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.11.0

Security for gpu-sentinel 0.1.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page