mmlearn

mmlearn aims at enabling the evaluation of existing multimodal representation learning methods, as well as facilitating experimentation and research for new techniques.

Quick Start

Installation

Prerequisites

The library requires Python 3.10 or later. We recommend using a virtual environment to manage dependencies. You can create a virtual environment using the following command:

python3 -m venv /path/to/new/virtual/environment
source /path/to/new/virtual/environment/bin/activate

Installing binaries

To install the pre-built binaries, run:

python3 -m pip install mmlearn

python3 -m pip install mmlearn[vision,audio]

Building from source

To install the library from source, run:

git clone https://github.com/VectorInstitute/mmlearn.git
cd mmlearn
python3 -m pip install -e .

Running Experiments

We use Hydra and hydra-zen to manage configurations in the library.

For new experiments, it is recommended to create a new directory to store the configuration files. The directory should have an __init__.py file to make it a Python package and an experiment folder to store the experiment configuration files. This format allows the use of .yaml configuration files as well as Python modules (using structured configs or hydra-zen) to define the experiment configurations.

To run an experiment, use the following command:

mmlearn_run 'hydra.searchpath=[pkg://path.to.config.directory]' +experiment=<name_of_experiment_yaml_file> experiment=your_experiment_name

Hydra will compose the experiment configuration from all the configurations in the specified directory as well as all the configurations in the mmlearn package. Note the dot-separated path to the directory containing the experiment configuration files. One can add a path to hydra.searchpath either as a package (pkg://path.to.config.directory) or as a file system (file://path/to/config/directory). However, new configs in mmlearn are added to hydra's external store inside path/to/config/directory/__init__.py which is only interpreted when the config directory is added as a package. Hence, please refrain from using the file:// notation.

Hydra also allows for overriding configuration parameters from the command line. To see the available options and other information, run:

mmlearn_run 'hydra.searchpath=[pkg://path.to.config.directory]' +experiment=<name_of_experiment_yaml_file> --help

By default, the mmlearn_run command will run the experiment locally. To run the experiment on a SLURM cluster, we use the submitit launcher plugin built into Hydra. The following is an example of how to run an experiment on a SLURM cluster:

mmlearn_run --multirun \
    hydra.launcher.mem_per_cpu=5G \
    hydra.launcher.qos=your_qos \
    hydra.launcher.partition=your_partition \
    hydra.launcher.gres=gpu:4 \
    hydra.launcher.cpus_per_task=8 \
    hydra.launcher.tasks_per_node=4 \
    hydra.launcher.nodes=1 \
    hydra.launcher.stderr_to_stdout=true \
    hydra.launcher.timeout_min=720 \
    'hydra.searchpath=[pkg://path.to.my_project.configs]' \
    +experiment=my_experiment \
    experiment_name=my_experiment_name

This will submit a job to the SLURM cluster with the specified resources.

Note: After the job is submitted, it is okay to cancel the program with Ctrl+C. The job will continue running on the cluster. You can also add & at the end of the command to run it in the background.

Summary of Implemented Methods

Contributing

If you are interested in contributing to the library, please see CONTRIBUTING.MD. This file contains many details around contributing to the code base, including are development practices, code checks, tests, and more.