fdq 0.1.13

ML runtime (https://pypi.org/project/fdq/)

FDQ | Fonduecaquelon

A fonduecaquelon is the heavy pot that keeps cheeses (e.g. 50% Gruyère and 50% Vacherin) melting smoothly into a perfectly blended whole — and FDQ does the same for deep learning. It keeps models, data loaders, training loops, and tools at a steady “temperature” so everything works seamlessly together, streamlining PyTorch workflows by automating repetitive tasks and providing a flexible, extensible framework for experiment management. Built for ML engineers who want to focus on experiments rather than boilerplate, FDQ lets you spend more time innovating and less time setting up.

• GitHub Repository
• PyPI Package

🚀 Features

• Minimal Boilerplate: Define only what matters — FDQ handles the rest.
• Flexible Experiment Configuration: Use YAML config files with Hydra composition and runtime overrides.
• Multi-Model Support: Seamlessly manage multiple models, losses, and data loaders.
• Cluster Ready: Submit jobs to SLURM clusters with ease using built-in utilities such as automatic job resubmission.
• Extensible: Easily integrate custom models, data loaders, and training/testing loops.
• Automatic Dependency Management: Install additional pip packages per experiment.
• Distributed Training: Out-of-the-box support for PyTorch DDP.
• Model Export & Optimization: Export trained models to ONNX with optimization options.
• High-Performance Inference: TensorRT integration for GPU-accelerated inference with up to 10x speedup.
• Model Compilation: JIT tracing/scripting and torch.compile support for optimized execution.
• Interactive Model Dumping: Intuitive interface for exporting and optimizing trained models.
• Monitoring Tools: Built-in support for Weights & Biases and TensorBoard.

🛠️ Installation

FDQ supports two ways to submit jobs to a Slurm cluster. The recommended option is to install the lightweight submit extra, which provides the fdq_submit command without pulling in PyTorch or the full ML runtime:

pip install fdq[submit]

To run/debug experiments, install the latest release from PyPI:

pip install fdq[full]

If you have an NVIDIA GPU and want to run inference, install GPU dependencies:

pip install fdq[gpu]

For development and the latest features, clone the repository:

git clone https://github.com/mstadelmann/fonduecaquelon.git
cd fonduecaquelon
pip install -e ".[dev,gpu]"

📖 Usage

Table of Contents

• Local Experiments
• SLURM Cluster Execution
• Model Export and Optimization
• Additional CLI Options

Local Experiments

All experiment parameters are defined in a config file. Config files can inherit from a parent / defaults file for easy reuse and organization.

Run an experiment locally:

fdq --config-path <path_to_config_files> --config-name <name_of_config_file>
# e.g.
fdq --config-path /home/marc/dev/fonduecaquelon/experiment_templates/mnist --config-name mnist_class_dense

SLURM Cluster Execution

Run experiments on SLURM by adding a slurm_cluster section to your config. See segment_pets_01.yaml.

When using chained config files, submit.py merges Hydra-style defaults entries before creating the SLURM submit script, so inherited mode, slurm_cluster, and store values are available to the submitter. Values in the launched child config override parent values.

Minimal example (YAML):

slurm_cluster:
  fdq_test_repo: false         # if true, installs fdq from test.pypi.org instead of PyPI (for pre-release versions)
  fdq_version: 0.1.13          # exact fdq version to install in the SLURM job environment
  python_env_module: "python/3.12.4"
  uv_env_module: "uv/0.6.12"
  cuda_env_module: "cuda/12.8.0"
  scratch_results_path: "/scratch/fdq_results/"   # temporary output dir on fast scratch disk; results are copied back on completion
  scratch_data_path: "/scratch/fdq_data/"         # if set, input data is copied here before training for faster I/O; omit or set null to skip
  log_path: "~/dev/fonduecaquelon/slurm_log"
  job_time: 15                 # maximum job runtime in minutes
  stop_grace_time: 5           # minutes between SIGTERM and SIGKILL; allows final checkpoint save and wandb upload on timeout
  cpus_per_task: 8
  gres: "gpu:1"
  mem: "20G"
  partition: gpu
  account: "cai_ivs"
  auto_resubmit: true          # if true, automatically resubmit and resume training when the job hits the time limit

When submitting jobs to a Slurm cluster, the only supported modes are:

mode:
  run_train: true|false
  run_test_auto: true|false

The remaining actions have to be run in an interactive session.

There are two ways to submit an experiment.

Option 1: installed command

fdq_submit /path/to/config.yaml

This is the recommended path when you can install either fdq[submit] or fdq[full] on the machine that submits the job. It keeps the command short and uses the same package entry-point mechanism as fdq.

Option 2: standalone script

Some clusters restrict package installation on login nodes. In that case, download the standalone submit script and run it directly with Python:

wget https://raw.githubusercontent.com/mstadelmann/fonduecaquelon/refs/heads/main/src/fdq/submit.py
python submit.py /path/to/config.yaml

Both commands use the same submit helper. The installed fdq_submit command is the cleaner option when installation is possible; the standalone script is a fallback for constrained login-node environments.

Parameter studies can be launched by marking scalar keys with @p. Numeric ranges use [start:stop:count], and categorical values use colon-separated entries. See mnist_class_dense_param_study.yaml for a complete example. Ranges are inclusive and multiple study parameters are submitted as a Cartesian product:

models:
  simpleNet:
    optimizer:
      args:
        lr@p: [0.001:0.005:5]

This submits runs with lr=0.001, 0.002, 0.003, 0.004, and 0.005. Values whose keys do not end in @p are regular config values, even if they look like lists or contain colons.

Categorical values can be written as colon-separated values on @p keys:

data:
  OXPET:
    args:
      shuffle_train@p: [true:false]

models:
  simpleNet:
    optimizer:
      class_name@p: ["torch.optim.Adam":"torch.optim.SGD"]

This submits all combinations of shuffle_train=true|false and class_name=torch.optim.Adam|torch.optim.SGD.

Notes:

• SLURM logs are written to slurm_log/.
• Results are organized under the configured store.results_path (when using submit.py on Slurm cluster, to scratch_results_path, which are then automatically copied back to store.results_path at job termination).

Model Export and Optimization

After training, export and optimize models for deployment:

# Interactive model dumping with export options (Hydra-style)
fdq --config-path <path_to_config_dir> --config-name <config_basename> mode.run_train=false mode.dump_model=true

This launches an interactive interface where you can:

• Export to ONNX: Convert PyTorch models to ONNX format using Dynamo or TorchScript
• JIT Compilation: Trace or script models with PyTorch JIT
• TensorRT Optimization: Compile models for GPU inference with FP32, FP16, or INT8 precision
• Performance Benchmarking: Compare optimized vs. original model performance

Additional CLI Options

You can overwrite all configurations at launch time (Hydra-style). This is mostly interesting to change the operations that you want FDQ to run:

# Run default (as defined in the mode section of the config file)
fdq --config-path <path_to_config_dir> --config-name <config_basename>

# Skip training
fdq --config-path <path_to_config_dir> --config-name <config_basename> mode.run_train=false

# Train and test automatically
fdq --config-path <path_to_config_dir> --config-name <config_basename> mode.run_train=false mode.run_test_auto=true 

# Interactive testing
fdq --config-path <path_to_config_dir> --config-name <config_basename> mode.run_train=false mode.run_test_interactive=true 

# Export and optimize models
fdq --config-path <path_to_config_dir> --config-name <config_basename> mode.run_train=false mode.dump_model=true 

# Run inference tests
fdq --config-path <path_to_config_dir> --config-name <config_basename> mode.run_train=false mode.run_inference=true 

# Print model architecture before training
fdq --config-path <path_to_config_dir> --config-name <config_basename> mode.run_train=true mode.print_model_summary=true 

# Resume from checkpoint
fdq --config-path <path_to_config_dir> --config-name <config_basename> mode.run_train=true mode.resume_chpt_path=</path/to/checkpoint>

🚄 Model Export & Deployment

FDQ offers full model export and optimization support for deployment:

Export Options

• ONNX Export: Convert models to ONNX for cross-platform use

  • Dynamo-based export for the latest PyTorch features
  • TorchScript export for broad compatibility
  • Automatic optimization and file size reporting

• JIT Compilation: PyTorch JIT tracing and scripting

  • Trace models for static graphs
  • Script models to preserve control flow
  • Automatic performance comparison with original models

• TensorRT Integration: GPU-accelerated inference with NVIDIA TensorRT

  • FP32, FP16, and INT8 precision
  • Automatic engine building and caching

Performance Features

• Automatic Benchmarking: Built-in performance testing with statistics
• Memory Optimization: Dynamic batch sizing and memory-efficient engines
• Cross-Platform: Compatible with various GPU architectures and CUDA versions

⚙️ Configuration Overview

FDQ uses YAML config files (with Hydra) to define experiments. These specify models, data loaders, training/testing scripts, and cluster settings.

Mode

You can either define in the config file what you want FDQ to do (train, test, resume training, dump, etc.), or you can specify/overwrite these parameters when launching the experiment (Hydra-style).

mode:
  run_train: true
  run_test_interactive: false
  run_test_auto: true
  dump_model: false
  run_inference: false
  print_model_summary: false
  resume_chpt_path: null

Models

Models are defined as dictionaries. You can use pre-installed ones (e.g. Chuchichaestli) or your own. Example:

models:
  myUNET:
    class_name: chuchichaestli.models.unet.unet.UNet

Access models in training via experiment.models["myUNET"]. The same structure applies to losses and data loaders.

Data Loaders

Your data loader class must implement create_datasets(experiment, args), returning:

return {
    "train_data_loader": train_loader,
    "val_data_loader": val_loader,
    "test_data_loader": test_loader,
    "n_train_samples": n_train,
    "n_val_samples": n_val,
    "n_test_samples": n_test,
    "n_train_batches": len(train_loader),
    "n_val_batches": len(val_loader) if val_loader is not None else 0,
    "n_test_batches": len(test_loader),
}

These values are available as experiment.data["<name>"].<key>.

DataLoader worker options

The example data preparators use the standard num_workers value from data.<name>.args for regular and DDP runs. In DDP, FDQ no longer rewrites this value; if num_workers > 0, FDQ prints a warning because multiprocessing DataLoader workers can occasionally contribute to DDP stalls or NCCL timeouts. If that happens, set num_workers: 0 in the dataset args.

prefetch_factor is optional and is only passed to PyTorch when num_workers > 0; when num_workers == 0, the preparators pass prefetch_factor=None because PyTorch requires that combination.

When num_workers > 0, the example preparators default persistent_workers to true and print a warning so worker processes are reused across epochs. Override it explicitly when needed:

data:
  OXPET:
    args:
      num_workers: 4
      prefetch_factor: 2       # optional; only used when num_workers > 0
      persistent_workers: false # optional override; default is true when num_workers > 0

Training Loop

Define a function in your training script:

def fdq_train(experiment: fdqExperiment):

Within it, you can access components:

nb_epochs = experiment.cfg.train.args.epochs
data_loader = experiment.data["OXPET"].train_data_loader
model = experiment.models["myUNET"]

See train_oxpets.py for an example.

At the beginning of each epoch call experiment.on_epoch_start() and at the end call experiment.on_epoch_end(...). These hooks reset per‑epoch timers/counters, aggregate metrics, and perform logging (TensorBoard / Weights & Biases) and any scheduling/checkpoint logic tied to epoch boundaries.

Minimal pattern:

def fdq_train(experiment: fdqExperiment):
    nb_epochs = experiment.cfg.train.args.epochs
    train_loader = experiment.data["OXPET"].train_data_loader

    for epoch in range(nb_epochs):
        experiment.on_epoch_start()

        running_loss = 0.0
        for batch in train_loader:
            # forward / loss / backward / optimizer step ...
            pass

        # Example scalar logging
        scalars = {"train/loss": running_loss / max(1, len(train_loader))}
        experiment.on_epoch_end(log_scalars=scalars)

See the full implementation in train_oxpets.py for a richer example (images, text, or additional metrics).

Testing Loop

Testing is similar. Define:

def fdq_test(experiment: fdqExperiment):

Automatic testing loads the validation-best model by default. Configure it with:

test:
  processor: /path/to/test_script.py
  test_model: best        # aliases: best_val, best_train, last

See oxpets_test.py for reference.

💾 Dataset Caching

FDQ includes a dataset caching system to speed up training by caching preprocessed data to disk and loading it into RAM. See segment_pets_13_cached.yaml for an example.

How It Works

1. Deterministic Preprocessing & Caching: Expensive transformations (resizing, normalization, data loading) are applied once and cached as HDF5 files.
2. On-the-fly Augmentation: Fast, random augmentations (e.g. flips, rotations) are applied during training.

Configuration

Enable caching in your config:

data:
  OXPET:
    processor: /path/to/data_preparator.py
    args:
      base_path: /path/to/data
      train_batch_size: 8
      val_batch_size: 8
      test_batch_size: 1
    caching:
      enabled: true           # set to false to disable caching without removing the config block
      cache_dir: /path/to/cache
      compress_cache: false   # compress HDF5 files on disk; saves space at the cost of slower reads
      pin_memory: true
      shuffle_train: true     # shuffle the cached training split each epoch
      shuffle_val: false
      shuffle_test: false

Cache files are written to cache_dir as HDF5 files. After a cache file is created or found, the cached split is loaded fully into RAM by CachedDataset. The RAM-backed cached DataLoaders always use num_workers=0 and persistent_workers=false; there is no separate caching worker option.

The first-time cache generation step still iterates the original dataset through a temporary DataLoader. It keeps the original DataLoader's num_workers, but reconfigures iteration to batch_size=1, shuffle=false, drop_last=false, pin_memory=false, and prefetch_factor=None so every sample is cached in a deterministic order.

Custom Augmentations

Define augmentations:

# oxpets_augmentation.py
def augment(sample, experiment=None):
    """Apply custom augmentations to cached dataset samples."""
    sample["image"], sample["mask"] = experiment.transformers["random_vflip_sync"](
        sample["image"], sample["mask"]
    )
    return sample

Reference in your config:

data:
  OXPET:
    caching:
      nondeterministic_transforms:
        processor: /path/to/oxpets_augmentation.py

See segment_pets_14_cached_augmentations.yaml for the full pattern.

Benchmarking Dataloader Performance

To measure the actual benefit of caching, the dataset preparator supports a slow option that adds an artificial ~10 ms delay per sample (a CPU matrix inversion), mimicking a slow or I/O-bound __getitem__. Once samples are served from cache, this overhead is bypassed entirely.

data:
  OXPET:
    args:
      slow: true  # add ~10ms artificial delay per sample to simulate a slow dataloader

Compare segment_pets_12_slow.yaml (slow, no cache) with segment_pets_13_cached.yaml (slow + cache) to quantify the improvement.

🧮 Mixed precision

Leveraging torch.amp for mixed precision training can dramatically accelerate your training workflow. For a practical implementation, see this example.

Observed speedup on H200sxm GPUs:

Experiment	Time per epoch [s]
segment pets with AMP	100
segment pets without AMP	170

🖧 Distributed Training

To run with PyTorch DDP, add:

slurm_cluster:
  world_size: 2
  cpus_per_task: 16
  gres: gpu:h200sxm:2

See segment_pets_02_dist2.yaml.

Use the same number of GPUs as your world size. DDP requires more CPU cores and memory, since multiple data loaders run in parallel. It's most beneficial for large models, as overhead is significant.

FDQ uses the same data.<name>.args.num_workers value for regular and DDP runs. In DDP, num_workers > 0 can be useful for slow input pipelines, but it creates workers per rank and can make hangs harder to diagnose. FDQ therefore leaves your value unchanged and prints a warning when DDP starts with num_workers > 0; set num_workers: 0 if you encounter DDP hangs or NCCL timeouts.

Observed speedup on H200sxm GPUs:

Experiment	Time per ep. w/o AMP [s]	with AMP [s]
segment pets default	170	100
DDP with 2 GPUs	100	65
DDP with 4 GPUs	60	45

By toggling mixed precision, you can directly observe how more intensive workloads see greater speedups when using DDP.

📦 Installing Additional Python Packages in SLURM

If your experiment requires extra packages, specify them in additional_pip_packages. FDQ installs them before execution.

Example (YAML):

slurm_cluster:
  fdq_version: 0.1.13
  # ... other settings ...
  additional_pip_packages:
    - monai==1.4.0
    - prettytable

🐛 Debugging

For debugging, install FDQ in development mode:

git clone https://github.com/mstadelmann/fonduecaquelon.git
cd fonduecaquelon
pip install -e ".[dev,full]"

Run the test suite from the repository root:

python -m pytest
ruff check .

VS Code Setup

1. Open your project in VS Code.
2. Add or update .vscode/launch.json to run run_experiment.py:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "FDQ Experiment Debug",
            "type": "debugpy",
            "request": "launch",
            "debugJustMyCode": false,
            "program": "${workspaceFolder}/src/fdq/run_experiment.py",
            "console": "integratedTerminal",
            "args": [
                "--config-path", "${workspaceFolder}/experiment_templates/segment_pets",
                "--config-name", "segment_pets_01"
            ],
            "cwd": "${workspaceFolder}"
        }
    ]
}

3. Debug/test your code.

📝 Tips

• Config Inheritance: Use Hydra’s defaults list in your YAML configs to include/extend base configs and reduce duplication.
• Multiple Models/Losses: Add multiple models and losses to config dictionaries as needed.
• Cluster Submission: submit.py handles SLURM job script generation, submission, environment setup, and result copying.
• Model Export: Set mode.run_train=false mode.dump_model=true for interactive model export and optimization.
• VRAM Estimation: At training startup FDQ automatically prints an estimated VRAM breakdown (parameters, gradients, optimizer state, and activations measured via a dummy forward pass) to help right-size your GPU request before submitting to the cluster.

📚 Resources

• Experiment Templates
• Chuchichaestli Models

🤝 Contributing

Contributions are welcome! Please open issues or pull requests on GitHub.

🧀 Enjoy your Fondue!

🧾 Changelog

• 0.1.13: Simplify DataLoader worker handling: remove ddp_num_workers, warn when DDP runs with num_workers > 0, keep cached RAM-backed loaders single-process, support optional prefetch_factor, and default persistent_workers=true when num_workers > 0 unless explicitly disabled.
• 0.1.11: DDP reliability improvements: VRAM estimation is now skipped in DDP mode to avoid rank desync from an asymmetric dummy forward pass on rank 0 only.
• 0.1.10: Allow specifying trained_model_path in a model's config block to load a checkpoint from a fixed path during test mode, without interactive prompting.
• 0.1.9: Fix crash when a model config block has no optimizer key (attribute access replaced with .get()).
• 0.1.8: Package fdq_submit as an installed console command, add lightweight fdq[submit] installation for Slurm login nodes, move the full ML runtime dependencies to fdq[full], and update CI/development installs to use the full extra for tests.
• 0.1.7: Various wandb bugfixes.
• 0.1.1 – 0.1.3: Parameter study support: numeric ranges ([start:stop:count]) and categorical values via @p-suffixed config keys; submissions are the Cartesian product of all study parameters.
• 0.0.75: Fix crash when no transforms are defined in config.
• 0.0.74: Configuration files switched from JSON to YAML, using Hydra in the backend for composition and runtime overrides.