nn-tilde 1.6.0

Set of tools to create nn_tilde compatible models.

Demonstration video

Installation

Grab the latest release of nn~ ! Be sure to download the correct version for your installation.

MaxMSP

Uncompress the .tar.gz file in the Package folder of your Max installation, i.e. in Documents/Max [your version]/Packages/. You can then instantiate an nn~ object! Alt-click the nn~ object to open the help patch, or access the nn~ Overview patch in the Extras menu.

Mac alert : codesigned with IRCAM identity and not trigger MacOS quarantine ; if it does so, please launch in the terminal :

cd "~/Max X/Packages/nn_tilde
sudo codesign --deep --force --sign - support/*.dylib
sudo codesign --deep --force --sign - externals/*/Contents/MacOS/*
xattr -r -d com.apple.quarantine externals/*/Contents/MacOS/*  

Alt+click on the nn~ object to open the help patch, and follow the tabs to learn more about this project.

PureData

Uncompress the .tar.gz file in the Package folder of your Pd installation, i.e. in Documents/Pd/externals/. You can then add a new path in the Pd/File/Preferences/Path menu pointing to the nn_tilde folder.

Similarly, the external should not be blocked on recent MacOS systems. It it still is, cd to the nn_tilde folder and fix with

xattr -r -d com.apple.quarantine Documents/Pd/externals/nn_tilde
sudo codesign --deep --force --sign - Documents/Pd/externals/nn_tilde/*.dylib
sudo codesign --deep --force --sign - Documents/Pd/externals/nn_tilde/nn\~.pd_darwin

Usage

Pretrained models

At its core, nn~ is a translation layer between Max/MSP or PureData and the libtorch c++ interface for deep learning. Alone, nn~ is like an empty shell, and requires pretrained models to operate. Since v1.6.0, you can download them directly through Forum IRCAM API. Alternatively, you can find a few RAVE models here or here. Few vschaos2 models are also availablehere.

Pretrained model for nn~ are torchscript files, with a .ts extension. You can add these files to nn_tilde/models folders, or any place accessible through Max / Pd filesystem (Max: Options/File Preferences, PureData: File/Preferences/Path).

New : since v1.6.0, some models are directly downloadable through IRCAM Forum API.

Once this is done, you can load a model with nn~ by providing its name as first argument (for example, here isis.ts located inside nn_tilde/models for Max, or among the PureData patch):  

Max / MSP	PureData

Model information fetching

Model information fetching

Coming with v1.6.0, the nn.info object allows model inspection and fetching avilable models for download on the IRCAM-API. With this object, you can get available methods and attributes for a given model. For example, you can see below that a RAVE model has three different methods : encode, decode, and forward.

Methods

Models can have several methods, that correspond to several processing pipelines the model can achieve. Hence, each method can have a different number in inlets / outlets. The method is given as the third argument (for exemple, decode above), and equals forward by default.

Attributes

It is possible the internal state of the module through attributes, that are model-dependent and defined at exportation. Model attributes can be set using messages, with the following syntax:

set ATTRIBUTE_NAME ATTRIBUTE_VAL_1 ATTRIBUTE_VAL_2

Using Max/MSP and PureData graphical objects, this can lead to an intuitive way to modify the behavior of the model, as shown below where we have two model attributes (i.e. generation temperature and generation mode), and the special enable attribute.

Max / MSP	PureData

New in 1.6.0

• Buffers (Max) / Array (Pd) attribute setting to allow the .ts model to access internal buffers / arrays.
• torch.Tensor attributes can be set through Max/MSP [array], allowing to set attributes of unlimited size.

Buffer configuration

Internally, nn~ has a circular buffer mechanism that helps maintain a reasonable computational load, if the given buffer size is greater tha 0. You can modify its size through the use of an additional integer after the method declaration, as shown below.

ImportantFor Windows users, the circular buffer is automatically disabled because of a memory leak that occurs when a TorchScript model is used in a separate thread. Unfortunately, this implies a much lower efficiency in terms of CPU.

Max / MSP	PureData

Multichannel (Max/MSP)

The Max/MSP release of nn~ includes additional externals, namely mc.nn~ and mcs.nn~, allowing the use of the multicanal abilities of Max 8+ to simplify the patching process with nn~ and optionally decrease the computational load.

In the following examples, two audio files are being encoded then decoded by the same model in parallel

This patch can be improved both visually and computationally speaking by using mc.nn~ and using batch operations

Using mc.nn~ we build the multicanal signals over the different batches. In the example above, each multicanal signal will have 2 different canals. We also propose the mcs.nn~ external that builds multicanal signals over the different dimensions, as shown in the example below

In the example above, the two multicanals signals yielded by the nn~ rave encode 2 object have 16 canals each, corresponding to the 16 latent dimensions. This can help patching, while keeping the batching abilities of mc.nn~ by creating an explicit number of inlets / oulets corresponding to the number of examples we want to process in parallel.

To recap, the regular nn~ operates on a single example, and has as many inlets / outlets as the model has inputs / outputs. The mc.nn~ external is like nn~, but can process multiple examples at the same time. The mcs.nn~ variant is a bit different, and can process mulitple examples at the same time, but will have one inlet / outlet per examples.

Lazy mode (Max/MSP)

Since v1.6.0, nn~ has a void mode, that allows to initialise it with a fixed number of inlets / outlets, and may be attached to a model afterwards. This can be done with the void special model, that enables this lazy initialisation.

Special messages

enable [0 / 1]

Enable / Disable computation to save up computation without deleting the model. Similar to how a bypass function would work.

reload

Dynamically reloads the model. Can be useful if you want to periodically update the state of a model during a training.

dump

Prints methods / attributes of the loaded model.

print_available_models

Prints models downloadable through API.

download

Download a model from the API.

delete

Deletes a downloaded model.

load

Change dynamically the incoming model.

method

Change dynamically the used method.

Build Instructions

macOS

• Download the latest libtorch (CPU) here and unzip it to a known directory
• Run the following commands:

git clone https://github.com/acids-ircam/nn_tilde --recurse-submodules
cd nn_tilde
curl -L https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh > miniconda.sh
chmod +x ./miniconda.sh
bash ./miniconda.sh -b -u -p ./env
source ./env/bin/activate
pip install -r requirements.txt
conda install -c conda-forge curl
mkdir build
cd build
mkdir puredata_include
curl -L https://raw.githubusercontent.com/pure-data/pure-data/master/src/m_pd.h -o puredata_include/m_pd.h
export CC=$(brew --prefix llvm)/bin/clang
export CXX=$(brew --prefix llvm)/bin/clang++
cd build
cmake ../src -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_PREFIX_PATH=../env/lib/python3.12/site-packages/torch -DCMAKE_BUILD_TYPE=Release -DPUREDATA_INCLUDE_DIR=../puredata_include -DCMAKE_OSX_ARCHITECTURES=arm64
cmake --build . --config Release

please replace arm64 in the last line by x86_64 if you want compile for 64 bits. You can remove -DPUREDATA_INCLUDE_DIR=../puredata_include to compile only for Max. The Max package is produced in src/, and Pd external in build/frontend/puredata/Release.

Windows

• Download Libtorch (CPU) and dependencies here and unzip to a known directory.
• Install Visual Studio Redistribuable
• Run the following commands (here for Git Bash):

git clone https://github.com/acids-ircam/nn_tilde --recurse-submodules
cd nn_tilde
curl -L https://download.pytorch.org/libtorch/cpu/libtorch-win-shared-with-deps-2.6.0%2Bcpu.zip > "libtorch.zip"
unzip libtorch.zip
mkdir pd
cd pd
curl -L https://msp.ucsd.edu/Software/pd-0.55-2.msw.zip -o pd.zip
unzip pd.zip
mv pd*/src .
mv pd*/bin .
cd ..
git clone https://github.com/microsoft/vcpkg.git
cd vcpkg 
./bootstrap-vcpkg.bat
./vcpkg.exe integrate install
./vcpkg.exe install curl
cd ..
mkdir build
cd build
mkdir puredata_include
curl -L https://raw.githubusercontent.com/pure-data/pure-data/master/src/m_pd.h -o puredata_include/m_pd.h
export CC=$(brew --prefix llvm)/bin/clang
export CXX=$(brew --prefix llvm)/bin/clang++
cd build
cmake ../src -G "Visual Studio 17 2022" -DTorch_DIR=../libtorch/share/cmake/Torch -DPUREDATA_INCLUDE_DIR=../pd/src -DPUREDATA_BIN_DIR=../pd/bin -A x64
cmake --build . --config Release

You can remove -DPUREDATA_INCLUDE_DIR=../puredata_include to compile only for Max. The Max package is produced in src/, and Pd external in build/frontend/puredata/Release.

Raspberry Pi

not availble in v1.6.0, planned in next version ; please take previous versions if needed

While nn~ can be compiled and used on Raspberry Pi, you may have to consider using lighter deep learning models. We currently only support 64bit OS.

Install nn~ for PureData using

curl -s https://raw.githubusercontent.com/acids-ircam/nn_tilde/master/install/raspberrypi.sh | bash

Funding

This work is led at IRCAM, and has been funded by the following projects

• ANR MakiMono
• ACTOR
• DAFNE+ N° 101061548