Skip to main content

Physical Simulations on Images.

Project description

Runtime-Guard

The RuntimeGuard provides functionality to make sure your algorithm runs fine. it got developed to counter memory exhaustion during training deep learning models. Does not need any dependencies!

It supports RAM, CPU, GPU, mean-loop time, watchdog timer, and memory leak checks.

> Documentation <


Installation

Just install it over PyPI, no dependencies needed:

pip install runtime_guard

Usage

Three ways to use it:

Standard:

import runtime_guard as run

guard = run.RuntimeGuard()

for epoch in epochs:
    for idx, x, y in data:
        guard.start_loop()

        # do work

        guard.update()

    if validate:
        guard.pause()
        # validation

Iterator-Method:

import runtime_guard as run

guard = run.RuntimeGuard()

for epoch in epochs:
    for idx, x, y in guard(data):
        # do work

    with run.Pause(guard):
        if validate:
            # or just: guard.pause()
            # validation

Context-Method:

import runtime_guard as run

guard = run.RuntimeGuard()

for epoch in epochs:
    for idx, x, y in data:
        with run.Run(guard):
            # do work

    with run.Pause(guard):
        if validate:
            # or just: guard.pause()
            # validation

Argument Parsing (Usage Part II)

import argparse

# Create a parser
parser = argparse.ArgumentParser()

#   example arguments from your application
parser.add_argument('--n_epochs', type=int, default=100, help='number of epochs with the initial learning rate')
parser.add_argument('--lr', type=float, default=0.0002, help='initial learning rate for adam')
parser.add_argument('--use_val_dataset', action='store_true', help='if true, creates a val dataset and evaluates at the end of each epoch')
parser.add_argument('--save_only_best_model', action='store_true', help='decides whether to save latest and best or only best model according to val loss (validation have to be used).')
#   arguments from RuntimeGuard
parser = runtime_guard.add_custom_args(parser)

# Get passed arguments
opt = parser.parse_args()

# Process arguments
if opt.use_runtime_guard:
    guard = run.RuntimeGuard(
        make_ram_check=opt.runtime_guard_make_ram_check,
        max_ram_usage_percentage=opt.runtime_guard_max_ram_usage_percentage,
        make_cpu_check=opt.runtime_guard_make_cpu_check,
        max_cpu_usage=opt.runtime_guard_max_cpu_usage,
        make_gpu_check=opt.runtime_guard_make_gpu_check,
        max_gpu_usage=opt.runtime_guard_max_gpu_usage,
        make_mean_loop_time_check=opt.runtime_guard_make_mean_loop_time_check,
        max_duration_factor_percentage=opt.runtime_guard_max_duration_factor_percentage,
        goal_mean_loop_time=opt.runtime_guard_goal_mean_loop_time,
        mean_loop_time_window_size=opt.runtime_guard_mean_loop_time_window_size,
        make_watchdog_timer_check=opt.runtime_guard_make_watchdog_timer_check,
        max_watchdog_seconds_timeout=opt.runtime_guard_max_watchdog_seconds_timeout,
        watchdog_seconds_waittime=opt.runtime_guard_watchdog_seconds_waittime,
        make_leak_check=opt.runtime_guard_make_leak_check,
        max_leak_mb=opt.runtime_guard_max_leak_mb,
        max_leak_ratio=opt.runtime_guard_max_leak_ratio,
        should_print=opt.runtime_guard_should_print,
        print_every_x_calls=opt.runtime_guard_print_every_x_calls,
        should_log=opt.runtime_guard_should_log,
        log_every_x_calls=opt.runtime_guard_log_every_x_calls,
        warm_up_iter=opt.runtime_guard_warm_up_iter,
        update_every_x_calls=opt.runtime_guard_update_every_x_calls,
        hard_exit=opt.runtime_guard_hard_exit
    )
else:
    guard = run.getDummyGuard()

Example Output

Runtime Guard Configuration
- RAM Check: True (max 56.29 GB)
- CPU Check: True (max usage 90.0%)
- GPU Check: True (max usage 90.0%)
- Mean Loop Time Check: True
    -> max_duration_factor_percentage = 3.0
    -> goal_mean_loop_time = 0.0
    -> mean_loop_time_window_size = 50
- Watchdog Timer Check: True
    -> max_watchdog_seconds_timeout = 300
    -> watchdog_seconds_waittime = 30
- Memory Leak Check: True
    -> max_leak_mb = 200.0
    -> max_leak_ratio = 0.2
- Print enabled: False
- Logging enabled: True (every 5000 calls, path='./runtime_guard.log')
- Warm-up iterations: 50
- Update every 50 calls
- Hard exit on failure: False


--------------------------------
Date: 18.12.2025
Time: 23:38
Loops: 5000
    - RAM checked
        -> usage = 13.23 GB / 56.29 GB (23.50%)
    - CPU checked
        -> usage = 6.00%
    - GPU checked (PyTorch)
        -> usage = 0.86 GB / 23.51 GB (3.65%)
    - Mean Loop Time checked
        -> mean time trend = 102.14% compared to the goal loop time
        -> current mean loop duration 0.01 / base mean loop duration 0.01
    - Watchdog is still fine!
    - Leak checked
        -> Growth = 0.03 MB
        -> Ratio  = 0.77%
        -> Top leak sources:
            /home/tippolit/anaconda3/envs/gan/lib/python3.8/site-packages/img_phy_sim/ray_tracing.py:1452: size=26.6 KiB (+8736 B), count=486 (+156), average=56 B
            /home/tippolit/anaconda3/envs/gan/lib/python3.8/tracemalloc.py:532: size=142 KiB (+6016 B), count=2656 (+118), average=55 B
            /home/tippolit/anaconda3/envs/gan/lib/python3.8/site-packages/torch/autograd/function.py:574: size=22.9 KiB (+3920 B), count=309 (+55), average=76 B
            /home/tippolit/anaconda3/envs/gan/lib/python3.8/site-packages/torch/nn/modules/module.py:1553: size=21.2 KiB (+1728 B), count=234 (+36), average=93 B
            /home/tippolit/anaconda3/envs/gan/lib/python3.8/site-packages/torch/nn/modules/container.py:219: size=14.0 KiB (+1728 B), count=212 (+36), average=68 B
            /home/tippolit/anaconda3/envs/gan/lib/python3.8/site-packages/torch/nn/parallel/comm.py:235: size=7872 B (+1472 B), count=123 (+23), average=64 B
            /home/tippolit/src/paired_image-to-image_translation/models/networks.py:536: size=9232 B (+864 B), count=106 (+18), average=87 B
--------------------------------

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

runtime_guard-0.1.tar.gz (14.0 kB view details)

Uploaded Source

File details

Details for the file runtime_guard-0.1.tar.gz.

File metadata

  • Download URL: runtime_guard-0.1.tar.gz
  • Upload date:
  • Size: 14.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.12.1

File hashes

Hashes for runtime_guard-0.1.tar.gz
Algorithm Hash digest
SHA256 51e9db21a6a2b102c147386f22a6ffe1f1444eaf752e661d92cd0b6cf41d99bc
MD5 748f9b5fe9462147570eb01807a82b3f
BLAKE2b-256 a73745c724438d66174c0e717d4523d90404a94a41f8d869b6e1e255cfcfabf5

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page