Milalib

Milalib is a library of helpers for Mila researchers to properly use the computing resources at their disposal.

All the listed functionality should work in slurm jobs on the Mila cluster. They may or may not work on DRAC clusters (I can report it doesn't work on Fir) -- you are welcome to try and report.

Monitoring GPU/resource usage

Monitoring resources is a little tricky for a few reasons:

  • nvidia-smi's GPU utilization reporting is very loose-grained, leading to overestimation
  • More precise profiling metrics such as sm_occupancy are exposed via DCGM
  • CPU/RAM measurements require knowing what was allocated to your job, which is hidden deep in /sys/fs/cgroup.

Milalib does all the above for you.

You can run the monitor standalone with uvx. It outputs one measurement per line in JSON format.

# Monitor continuously, sampling every 3 seconds (default is 5 seconds)
uvx milalib monitor -i 3

# Get a single sample and quit
uvx milalib monitor -s

# List all available metrics
uvx milalib monitor -l

# Monitor some metrics specifically:
uvx milalib monitor -m sm_occupancy -m gpu_util

# Pipe to jq to only get the numeric value
uvx milalib monitor -m sm_occupancy | jq .value

# Only output metrics for the first GPU
uvx milalib monitor -d gpu0

# Only output GPU metrics
uvx milalib monitor -d 'gpu.*'

You may also install milalib with pip install milalib or uv add milalib.

Wandb integration

If you use wandb for experiment logging, you can also log the monitoring metrics:

from milalib.wandb import monitor

watcher = monitor(
    interval=5,
    metrics=["gpu_util", "power"],
    devices=["gpu0"],
    prefix="milalib_",  # optional prefix
    log=wandb.log,  # optional, defaults to wandb.log
)
...  # train
watcher.stop()

Or, scoped to a block:

with monitor(interval=5, ...):
    ...  # train

Interface

milalib exposes an async generator for the metrics, streaming objects of type MetricValue. Use it like this:

import asyncio

from milalib.monitor.poll import MetricRequest
from milalib.monitor.stream import stream_metrics


async def main():
    request = MetricRequest(
        runner=directory.here().make_runner(),
        metrics=["sm_occupancy"],
    )
    async for value in stream_metrics(request, interval=0.1):
        print(value.timestamp, value.device, value.name, value.statistic, value.value)


asyncio.run(main())

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for breuleux from gravatar.com breuleux

Credits

Author: Olivier Breuleux

Requires

Python >=3.11

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

milalib-0.0.1.tar.gz (16.6 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

milalib-0.0.1-py3-none-any.whl (21.2 kB view details)

Uploaded Python 3

File details

Details for the file milalib-0.0.1.tar.gz.

File metadata

  • Download URL: milalib-0.0.1.tar.gz
  • Upload date:
  • Size: 16.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for milalib-0.0.1.tar.gz
Algorithm Hash digest
SHA256 227eb9258d24e2beb6daecc323a336a1db0bfcd865028bda11fdc91edc69d6a0
MD5 df76f2b53a76ace1ef15805038583f05
BLAKE2b-256 c8ab5038f526b640a433458c035aee237da6e9dc45f35eda71892ef011e89cc8

See more details on using hashes here.

Provenance

The following attestation bundles were made for milalib-0.0.1.tar.gz:

Publisher: publish.yml on breuleux/milalib

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file milalib-0.0.1-py3-none-any.whl.

File metadata

  • Download URL: milalib-0.0.1-py3-none-any.whl
  • Upload date:
  • Size: 21.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for milalib-0.0.1-py3-none-any.whl
Algorithm Hash digest
SHA256 5626660281f7f3eec9f1696d7c003a38844c2a7cdcd7cdc354f821f7df2cdaf5
MD5 203fa563b30da5d159093b0dd42ef9fa
BLAKE2b-256 991d7827bb8040b8b047f8155c875882ffb9ab1e8188a37e6bd0a49ca2108210

See more details on using hashes here.

Provenance

The following attestation bundles were made for milalib-0.0.1-py3-none-any.whl:

Publisher: publish.yml on breuleux/milalib

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page