fugue

fugue trains many neural networks on a single GPU and measures what you gain depending on how you run them. You declare a workload — an ensemble, a cross-validation, a hyperparameter search, a heterogeneous set of models — choose a concurrency strategy, and it leaves the time, energy, GPU utilisation and peak memory of that run in a CSV.

The name comes from the musical fugue: several independent voices enter on the same subject and sound at once, each one complete, over a single instrument.

Context

fugue is the experimental tool of the undergraduate thesis Desarrollo de estrategias para el entrenamiento paralelo de modelos neuronales en GPUs individuales — Licenciatura en Ciencias de la Computación, Facultad de Ciencias Exactas, Ingeniería y Agrimensura, UNR.

• Author: Joaquín Arroyo
• Advisor: Dr. Matías Gerard · Co-advisor: Dr. Leandro Vignolo

The document — methodology, results, analysis and conclusions — lives in a separate repo: thesis. This README covers only how to use the tool.

Why

The common case in a lab is not one huge model but many small ones: five folds, ten seeds, twenty hyperparameter configurations. None of them fills the GPU on its own. Running them one at a time wastes the card; running them all at once sometimes saturates it and ends up worse. Which of the two happens depends on the model, the batch size, the host RAM and the specific GPU — you don't derive it, you measure it.

fugue exists to measure that on your hardware, with the four strategies running the same workload under the same conditions.

It helps if you have one GPU and many independent trainings to launch. It does not help you split one large model across several GPUs: that is DDP/FSDP and a different problem.

Goal

Compare, over the same measurement, four ways of occupying a single GPU:

The comparison rests on measurement invariants: same GPU, same software environment, same data and same seeds for every strategy. The execution environment (torch, CUDA, driver and CPU versions) is stored in every result row, next to the numbers it qualifies.

Built-in executors

Each executor is a different way of dividing the GPU among the jobs of a workload. They run the same workload and write the same metrics, so comparing them is direct. What follows is the minimum needed to pick one and know what to expect; the full treatment — what each mechanism measures, how they behave and why — is in the thesis (work in progress).

seq — Sequential

Runs one job at a time, in the same process, until all are done. There is no concurrency: it is the denominator the other three are compared against. Each job gets its own seed derived from --seed.

mp — Concurrent processes with CUDA MPS

Launches jobs as independent processes (spawn) that share the GPU through CUDA MPS: their kernels coexist on different SMs instead of taking turns. The executor brings the nvidia-cuda-mps-control daemon up and down on its own, and assigns each client a fraction of the SMs via CUDA_MPS_ACTIVE_THREAD_PERCENTAGE, split evenly among the concurrent processes.

The cost is that each process pays for its own CUDA context, its own copy of the dataset and, with --compile, its own compilation.

Requires nvidia-cuda-mps-control to be installed and reachable on PATH (on several distributions it lives in /usr/sbin). setup.sh warns if it is missing.

ray — Ray Tune

Launches jobs as Ray actors, or as Ray Tune trials when the workload is a hyperparameter search. It does not use MPS: each worker opens its own CUDA context and the driver time-multiplexes them, switching contexts rather than dividing SMs.

What it contributes is not the sharing mechanism but the scheduler: --scheduler accepts fifo, asha, median and hyperband, and the last three cut off the worst-performing trials early. In HPO that usually matters more than any concurrency gain; outside HPO — ensembles, cross-validation — there is nothing to prune and only the coordinator's overhead remains.

unified_model — Unified Model

--exec unified_model trains every job inside a single PyTorch process, sharing the CUDA context and one copy of each dataset. It is an in-house implementation inspired by the UnifiedNN paper (Taki et al. 2024), with three extensions that can be switched off one at a time for ablation studies:

All three are on by default. --unified-model-baseline turns them off — the paper-equivalent configuration — and then they are switched back on one at a time. The effective combination is persisted in the features column (e.g. vf:1,s:1,fl:0).

To measure them one by one, --sweep ablation runs the six canonical configurations (baseline → vmap → two terminal branches, streams and compile, each with and without fused loss) instead of a single one. The flags and the ladder are declared by the executor, not the CLI: a new executor with its own knobs publishes them in its FeatureSet and they show up on their own in fugue run --help.

Installation

Python 3.12 or 3.13, and ~30 GB of disk. Two supported scenarios, and only two:

The CUDA Toolkit is not needed: PyTorch wheels ship their own CUDA runtime. setup.sh detects the system and resolves both paths — apt-get on Linux, brew on macOS — and on macOS installs the CPU stack and skips everything NVIDIA-related instead of failing.

Windows is not supported and there are no plans for it to be.

git clone https://github.com/joaquinarroyo/fugue.git
cd fugue
python3 -m venv venv && source venv/bin/activate
./setup.sh

Install it in a virtualenv of its own, not next to your project. The dependencies are pinned to exact versions — torch==2.11.0, ray==2.53.0 — and that is the point rather than an oversight: a speedup only means something if every strategy was measured against the same stack, so the stack is part of the instrument. Those pins will fight anything else you have installed. fugue is a measuring tool you run, not a library you build on.

Without cloning

python3 -m venv venv && source venv/bin/activate
pip install fugue-bench          # the distribution is fugue-bench
fugue --help                     # the command stays fugue

The distribution is fugue-bench because fugue on PyPI is an unrelated distributed-computing project. The import name and the CLI are both fugue; only the install line differs.

That is the way in if you want to run fugue — the CLI, the experiment registry, figures from CSVs that were already measured. What it does not do is the thing setup.sh exists for: pip takes PyTorch from PyPI, which is the CUDA build that wheel happens to target rather than the one matching your driver. On a machine where the numbers are meant to mean something, clone and run setup.sh.

setup.sh detects the CUDA version the driver reports, picks the matching PyTorch wheel (cu118 … cu130, or CPU if there is no GPU) and installs the project in editable mode. It exists only for that: the extra index cannot be declared in pyproject.toml.

To check everything landed:

nvidia-smi                                                    # driver alive
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
fugue validate                                                # every experiment resolves

Usage

The installation leaves a single command, fugue, and everything else hangs off it as a sub-command. fugue --help lists them; fugue <subcommand> --help shows the options of each.

Typical flow:

fugue list                         # what experiments exist
fugue validate                     # that models, datasets and profiles resolve
fugue profile                      # optional: profile the GPU (~30-60 min, once)

fugue run --exp E01 --exec seq     # baseline
fugue run --exp E01 --exec mp      # MPS
fugue run --exp E01 --exec ray --scheduler asha
fugue run --exp E01 --exec unified_model --compile

fugue plots                        # figures + tables for everything run so far

Main fugue run flags:

Concurrency: estimated, or by hand

How many jobs run at once and with how many DataLoader workers is estimated from the measured VRAM/RAM profiles and the host's cores, and is recorded in the max_parallel column of the group CSV.

The profile is optional. If there is none for that (model, dataset) pair — a brand-new GPU, your own model — there is nothing to divide memory by, so fugue warns and launches every job together. It may fit, or it may OOM: no check is possible without having measured first. That is where --max-parallel comes in, fixing the group size by hand so you can lower it until it fits:

fugue run --exp E02 --exec mp                    # all 7 at once, no safety net
fugue run --exp E02 --exec mp --max-parallel 3   # three at a time

fugue profile is still what turns that trial and error into an estimate: once the pair is measured, concurrency follows on its own and --max-parallel becomes unnecessary.

Campaigns

fugue run accepts several experiments and several configurations at once: the product of what you pass is the run matrix.

fugue run --exp E01:E15 --exec seq                   # full baseline
fugue run --exp E08:E10 --exec mp --compile          # only E08–E10, compiled
fugue run --exp 1,4,5 --exec seq mp ray              # three executors, three experiments
fugue run --exp E12:E15 --exec ray --scheduler fifo asha --compile both
fugue run --exp all --exec unified_model --sweep ablation --passes 3 --resume

• Experiment selection takes ids (E01 E03), inclusive ranges (E08:E10, E08:, :E05), lists (1,4,5) or all.
• --compile both runs each cell with and without torch.compile; --scheduler only expands for Ray, so sweeping it does not duplicate runs of the others.
• --passes N repeats the whole matrix; --resume skips cells already present in master_groups.csv (with --since TIMESTAMP to ignore older campaigns), and --retries N retries a failing cell.

Each cell runs in its own process, as the bash scripts this replaced used to do. That is not an implementation detail: the TorchInductor cache is cleared when the process starts, so --compile warmup is measured cold, the CUDA context starts clean and the allocator does not arrive fragmented from the previous run. It is also what makes it possible to kill and retry a hung cell — a watchdog declares one dead if it stops consuming CPU and GPU.

Configuration through the environment

Everything has a working default; these variables exist to run fugue outside the original machine without touching code.

FUGUE_DATA_DIR=/mnt/datasets FUGUE_OFFLINE=1 fugue run --exp E01 --exec mp

What it produces

A result is one row, not a directory:

runs/
├── master_individual.csv   # one row per job:   time, throughput, energy, final accuracy
├── master_groups.csv       # one row per group: makespan, total energy, peak VRAM, environment
└── samples/                # optional: GPU time series, if sampling was on

figs/2026-08-19_15-40-21/   # one directory per render
├── base_compile/ …         # speedup, wall-clock, energy, utilisation, VRAM
└── table_*.tex             # LaTeX tables (booktabs)

The most-used columns: executor, compile, exp_id, T_group (group makespan), throughput_group, energy_group, vram_peak_mb, max_parallel. The full schema of both CSVs is in AGENTS.md.

Speedup is not stored: it is derived when plotting, as seq's T_group over that of the compared strategy.

The CSVs that come with the repo are the thesis's own measurements — 504 groups over 17 experiments, on a T4 and an L4, in August 2026 — not reference numbers for your hardware. They are versioned so the document can be re-rendered without re-running a campaign that takes days. See runs/README.md for what was measured and how to read it. They are not part of the installed package: a fresh install writes its runs to the working directory instead.

Figures and tables

fugue plots draws what is in the CSVs: one comparison for non-HPO workloads and another for HPO, over the experiments that actually show up — not over a fixed list in the code. fugue tables generates only the tables and fugue plotter lets you pick executors, experiments and grouping by hand.

fugue plots                                    # everything there is
fugue plots --comparison base --lang en
fugue plots --experiments E01 E02 E03          # only these
fugue plots --comparison ablation              # extension study (opt-in)
fugue plots --comparison quality               # accuracy parity (opt-in)

The last two answer questions specific to this thesis — how much each Unified Model extension contributes, and whether accuracy holds — so they have to be asked for by name: they are not in the default render.

Experiments

17 experiments ship with the tool (src/fugue/helpers/experiments.py), in four families:

They range from minimal workloads — an MLP over CalHousing, where scheduling overhead dominates — to workloads that saturate the card: E16 and E17 replicate the UnifiedNN paper's groups, and E17 needs ≥ 24 GB of VRAM (it does not fit on a T4).

fugue list prints them with their hyperparameters.

Running something that is not in the registry

You do not need to define an experiment to try one: fugue run accepts the description directly, builds the definition in memory and runs.

fugue run --model resnet18 --dataset cifar10 --k 5 --epochs 10 --exec mp
fugue run --model mlp --dataset cal_housing --task regression --type cv --k 5
fugue run --model resnet18 --dataset cifar10 --type hpo \
          --search-space lr=0.1,0.01 batch_size=64,128 --exec ray
fugue run --type multi_model --model lenet resnet18 --dataset mnist cifar10

The id recorded in the CSVs is adhoc unless you pass --name. The results are rows like any other: same schema, same figures. To make something permanent it is worth writing it into registry/experiments.json, which is merged over the built-in catalogue — or pointing FUGUE_REGISTRY_DIR at your own copy.

Before a long run it is worth prefetching the datasets:

fugue prefetch   # the catalogue's seven datasets, including wikitext, sst2 and imagenette

Extending

Everything the tool knows about lives in a registry and is declared with a decorator: models, datasets, workload types and executors. External code uses the same decorators as the built-ins and never imports the core.

Models and datasets

# my_lab/extras.py
from helpers.extensions import DatasetBundle, register_model, register_dataset

@register_model("my_net")
def build(n_classes, in_shape):
    ...                                   # -> nn.Module

@register_dataset("my_data")
def load(root, download):
    ...                                   # -> DatasetBundle(train, test, n_classes, in_shape)

FUGUE_PLUGINS=my_lab.extras fugue validate
fugue profile --filter my_net__my_data   # optional: without this concurrency is not estimated

An installed package can declare a fugue.plugins entry point instead of using the environment variable.

Executors and workload types

An executor is declared once, with its slug, and everything else asks: the slug is what --exec accepts, what is stored in the CSV's executor column, and where the colour and label of figures and tables come from.

from core.executors import ExecutionStrategy, register_executor

@register_executor("mine", label="Mine", color="#8172B2")
class MyExecutor(ExecutionStrategy):
    def execute(self, jobs, gpu_index, seed):
        ...                               # -> (results, group_metrics)

A workload type — how an experiment expands into jobs — is registered the same way, under the value the experiment's type field will carry:

from core.workloads import WorkloadStrategy, register_workload

@register_workload("my_sweep")
class MySweep(WorkloadStrategy):
    def generate_jobs(self, exp_id, **kwargs):
        ...                               # -> [JobSpec, ...]

An executor's declarative flags (baseline, group_gpu_metrics, supports_scheduler, features_target, records, options) are what avoid the if executor == "ray" scattered through the code. Ablation ladders are not a separate flag: they come from the sweeps of the FeatureSet that features_target declares.

Tasks and optimizers

A task defines its loss, how the output is shaped before that loss, and what number it reports as "accuracy" — a hit rate, an R², a per-token accuracy:

from helpers.tasks import TaskSpec, register_task

register_task(TaskSpec(
    slug="my_task",
    criterion=MyLoss,
    prepare=lambda out, y: (out, y),
    evaluate=lambda model, loader, device: {"loss": ..., "acc": ...},
))

An optimizer is registered with both implementations, because the project builds it twice: once per job (plain torch) and once stacked over K sub-models in the Unified Model. Registering them together is what prevents it existing for three executors and missing from the fourth:

from helpers.optimizers import OptimizerSpec, register_optimizer

register_optimizer(OptimizerSpec(
    slug="lion",
    build=lambda params, lr, **kw: Lion(params, lr=lr, **kw),
    stacked_target="my_lab.stacked:build_stacked_lion",   # optional
))

Without stacked_target, the Unified Model rejects that optimizer by name rather than guessing. Same for a task with no vectorised kernel.

Development

pre-commit install           # once
pre-commit run --all-files   # ruff + mypy + assorted checks
pytest tests/                # 600+ tests, run on CPU

CI runs ruff, mypy and pytest on every PR, over CPU torch and with a minimal subset of dependencies (no ray, sklearn, transformers): every module that uses them has to import them inside the function that needs them. The test job fails if coverage drops below 40 %, as a ratchet and not as a target — a good part of the code needs a real GPU to execute.

Code conventions and architecture details: AGENTS.md.

Parked work

The saturation sweep and the theoretical-limit analysis — analytical ceiling from VRAM and CPU, detect_knee() over the throughput curve, the monitor's mem_bw_util_pct — were explored and left out; the thesis lists them as future work. They are not on main, they are on the archive/theoretical-limit tag:

git checkout -b saturation archive/theoretical-limit

A tag and not a branch on purpose: it is ~1000 lines that work but that nobody is carrying forward, and a parked branch suggests otherwise.

Common problems

nvidia-smi fails after a kernel update. DKMS did not rebuild the NVIDIA modules for the new kernel; dkms status shows which ones they are built for.

sudo apt-get install -y linux-headers-$(uname -r)
sudo /usr/sbin/dkms install nvidia-current/550.163.01 -k $(uname -r)
sudo modprobe nvidia

Ray kills workers over memory. The RAM profiles went stale, or the current GPU was never profiled. Compare with fugue simulate and re-profile with fugue profile.

MPS does not start, or is left as a zombie. A previous run left the daemon hanging:

sudo pkill -f nvidia-cuda-mps && sudo rm -rf /tmp/mps_pipe /tmp/mps_log

torch.compile fails. If the error mentions ptxas or Triton, reinstall torch with its official wheel, which ships its own ptxas. If it mentions nvcc or cpp_extension, then the full CUDA Toolkit is needed. If it is a RAM spike while compiling, re-profile with --compile so that ram_mb_compiled gets measured.

License

MIT — see LICENSE.