neural-tilde 0.0.1

Exporting PyTorch neural audio synthesis models in ExecuTorch for neural_tilde.

neural: neural audio synthesis in Max/MSP

Max externals for running PyTorch neural synthesis models in realtime/offline.
Models are exported from PyTorch with ExecuTorch and run inside Max with hardware acceleration on CPU / GPU / ANE (Apple Neural Engine).

This package has two families of Max objects:

Live models: streaming realtime DSP models (e.g. neural vocoder, neural codecs...):

object	use it for
neural.live~	a single model instance on one audio stream
mc.neural.live~	the same model broadcast across the channels of an mc signal
mcs.neural.live~	a fixed batch of streams processed in one pass (takes a batch-size argument)

Generative models — one-shot offline generators (e.g. latent-diffusion text-to-audio, audio-to-audio):

object	use it for
neural.gen~	run a generative model and write the output into a buffer~

Utility:

object	use it for
neural.tokenizer	turn a text prompt into token ids and attention mask
neural.gaussianize	map uniform jit.noise to Gaussian noise

Supported models:

• [Live] RAVE (Realtime Audio Variational autoEncoder)
• [Gen] Stable Audio 3 (text-to-audio, audio-to-audio)
• [Gen] CodiCodec-Flow

Note: Currently only available for MaxMSP on Apple Silicon, macOS. For Windows/CUDA, it's on the todo list.

Acknowledgement: neural.live~ and its mc, mcs variants reused an extensive amount of code from nn~, migrated from TorchScript/libtorch to more modern ExecuTorch. nn~ is the work by Antoine Caillon & Axel Chemla--Romeu-Santos (acids-ircam), licensed CC BY-NC 4.0.

How It Works

• Export: Export a PyTorch neural audio model using ExecuTorch, with a target hardware backend (see below). This results in a .pte (the runnable ExecuTorch program + weights) file.
• Config: Create a .json sidecar that details inlet/outlet, ratio, attributes, condition shapes, etc.
• Load: Hand neural.live~ / neural.gen~ the .pte and .json files in Max.

Supported back-ends (chosen when exporting the model):

• Core ML — Leverage Apple Neural Engine (ANE) for hardware acceleration. Needs macOS 15+ at runtime.
• MLX — Apple-Silicon GPU; Good for one-shot / buffer processing.
• XNNPACK — CPU inference with optimized kernels.
• portable — plain, unoptimized C++ kernels, maximum compatibility
• CUDA - Windows CUDA in progress

Install the externals

Download the pre-released externals. To install:

1. Unzip and copy the neural_tilde folder into your Max Packages folder, e.g. ~/Documents/Max 9/Packages/.
2. Restart Max. The streaming objects (neural.live~, mc.neural.live~, mcs.neural.live~) and the generative objects (neural.gen~, neural.tokenizer) are now available.

If macOS quarantines the externals (objects fail to instantiate), clear the quarantine flag or ad-hoc re-sign them. In Terminal, type:

cd ~/Documents/Max\ 9/Packages/neural_tilde
xattr -dr com.apple.quarantine externals/*.mxo
# or:  codesign --force --deep -s - externals/*.mxo

Use a pretrained model

Make sure to add your model's path in Max: The .pte and its .json should have the same filename. Put them together in a folder. In Max, open "Options/File Preferences", click "Add Path" on the bottom left corner, add your model's folder. In Max, create an external using:

[neural.live~ mymodel forward]

APIs:

• Arguments: neural.live~ <model.pte> <method> [buffer size]. The method defaults to forward; an autoencoder might expose encode / decode instead (one object per method). The .pte extension is optional.

• Inlets / outlets are created from the sidecar labels — one signal inlet per input channel, one signal outlet per output channel.

• Turn computation on: the enable_model attribute defaults to off, so the object outputs silence until you send it:

  enable_model 1
  

• Buffer size is adopted from the model's exported size (fixed-shape programs), so it overrides the optional box argument.

Useful messages / attributes:

message / attribute	effect
enable_model 0/1	disable / enable inference (default 0)
reload <path>	load a different model at runtime
get_methods	print the methods available in the loaded .pte
@<name> <v> / <name> <v>	the model's settable scalar controls, one Max attribute per sidecar attributes entry (e.g. @temperature 0.8), settable as box-args, in the inspector, or by message; clamped to the model's range and fed to the model every block. The sidecar description is the inspector label.
get <name> / get_attributes	print one control's value / list them all with values

Use mc.neural.live~ to run one model across every channel of an mc. patch cord, or mcs.neural.live~ to process a fixed batch of streams in a single forward pass.

Generative models (Stable Audio 3)

Some models aren't streaming DSP at all — they're one-shot generators: hand them a text prompt and they produce a fixed clip in a single (multi-second) pass. Stable Audio 3 (SA3) is a latent-diffusion text→audio model of this kind. Two objects handle it:

• neural.gen~ — a generic latent-diffusion runner. It loads the model .pte, takes its conditioning on one inlet per condition input (token ids, attention mask, and any negative-prompt pair), samples the model's noise itself, runs the generation on a worker thread (the audio thread never blocks), writes the result into a named buffer~, and bangs a done outlet.
• neural.tokenizer — turns a prompt into input_ids + attention_mask, emitting each as a single-key dictionary on its two outlets. Keeping it separate means neural.gen~ stays generic (a model with a different tokenizer, or none, just wires a different source into its inlets), and a second tokenizer can feed a negative prompt.

The model bundle

Put these four files in one folder and add it to Max's Options → File Preferences (the .pte is large and user-supplied, ~2.9 GB for SA3, so it is not shipped with the package):

file	what it is
full_pipeline_T32.pte	the ExecuTorch program + weights
full_pipeline_T32.json	model sidecar — typed inputs/output (no tokenizer settings)
full_pipeline_T32.tokenizer.json	the HuggingFace fast tokenizer
full_pipeline_T32.tokenizer.config.json	tokenizer settings neural.tokenizer reads

The SA3 export session emits all four (see its export/emit_sidecars.py). The protocol is in EXECUTORCH_PROTOCOL.md §7.

Patch it up

[prompt lofi house loop]              <- message to the tokenizer
       |
[neural.tokenizer full_pipeline_T32.tokenizer.config.json]
       |        |     <- outlet 0 = {input_ids}, outlet 1 = {attention_mask}
       |        |
[neural.gen~ full_pipeline_T32 @seconds_total 3 @seed 0]
       |               (inlet 0 = input_ids, inlet 1 = attention_mask)
   (done bang)        ... writes into [buffer~ gen 132300 2]

1. Create [buffer~ gen 132300 2] (stereo; SA3 outputs ~2.97 s at 44.1 kHz). Send neural.gen~ the message set gen to target it.
2. Send the prompt to neural.tokenizer (prompt lofi house loop); wire its two outlets into the matching neural.gen~ condition inlets (each is a single-key dict matched by name, so the order you wire them doesn't matter). neural.gen~ caches them.
3. Send generate (or bang) to neural.gen~. After a few seconds the done outlet bangs, the buffer is filled, and you can play it with groove~ gen / play~.

You can pass neural.tokenizer the *.tokenizer.config.json (it self-configures), or the bare *.tokenizer.json (it auto-looks for the matching *.tokenizer.config.json beside it, and warns if none is found and it falls back to defaults).

APIs

neural.gen~ — neural.gen~ <model.pte> [method] [buffer]:

message / attribute	effect
(condition inlets)	one inlet per condition-role input, in sidecar order (e.g. input_ids, attention_mask, and for a CFG model neg_input_ids, neg_attention_mask). Each accepts a dictionary (matched by key name — typically from neural.tokenizer) or a bare list (assigned to that inlet's condition by position). Cached; does not auto-run. The leftmost inlet also takes the control messages below. A condition left unsupplied is zero-filled at generate (a zero mask ⇒ unconditional) with a warning.
generate / bang	start a generation (ignored if one is already running)
set <buffer>	target buffer~ for the output
@<name> <v> / <name> <v>	the model's scalar controls, one attribute per sidecar attribute-role input (e.g. @seconds_total 3), settable as box-args, in the inspector, or by message; clamped to the model's range. Each attribute's sidecar description is shown as its inspector label.
init <buffer> / init	audio-to-audio: point an audio_init-role model at a source buffer~ to vary from (the host resamples / channel-maps / crops it to the model's geometry); init with no arg clears it (generate from silence)
get_attributes / get <name>	list the model's attributes + values / print one
seed <n>	RNG seed for the generation noise (reproducible)
reload <path>	load a different generative model
get_methods	print the methods available in the loaded .pte

Audio-to-audio. A sibling SA3 model bundle (audio2audio_T32.*) adds an init_audio input: send init <buffer~> to vary an existing clip, steered by the prompt. Its strength is the @init_noise_level attribute (0–1; 1 ignores the init and reduces to text-to-audio). The patch is otherwise identical to text-to-audio above.

neural.tokenizer — neural.tokenizer <tokenizer.json | *.config.json>:

message / attribute	effect
prompt <text…>	tokenize the text and output two single-key dictionaries — outlet 0 = {ids_key}, outlet 1 = {mask_key}
reload <path>	load a different tokenizer / config
@max_length 256	pad/truncate length
@pad_token <pad> @padding_side right	padding token and side
@ids_key input_ids @mask_key attention_mask	keys for the two single-key output dictionaries (= the model's condition input names)

A config file sets these attributes for you; a live @attr/message still overrides.

Negative prompt (classifier-free guidance). A CFG model bundle (*_cfg_T32.*) adds neg_input_ids + neg_attention_mask condition inputs and a @cfg_scale attribute (1 = no guidance; ~7 = stronger prompt adherence). Wire a second neural.tokenizer configured with @ids_key neg_input_ids @mask_key neg_attention_mask into those inlets; an empty negative prompt = unconditional. Leave it unwired and neural.gen~ zero-fills the negatives (also unconditional).

Notes. The ~2.9 GB weights load lazily on the first generate, so the first run has extra warm-up latency; later runs are fast. Generation runs off the audio thread, so DSP keeps playing throughout. A buffer~ carries Max's project sample-rate label while the data is 44.1 kHz — if your project runs at a different rate, the data is correct but plays back at the wrong pitch (resample, or run the project at 44.1 kHz).

Export your own neural synthesis model

Models are produced with the helper in python_tools/. It provides the neural_tilde Python module. Install it and the export back-ends by:

pip install -e .  
# Install the `neural_tilde` module, cached_conv, executorch, coremltools

Streaming models for neural.live~

See example codes below:

• Subclass neural_tilde.LiveModule,
• Build your network (for a click-free streaming convolutional net, one option is cached_conv; any stateful mechanism works),
• Use register_method(...) to describe each method's channels / ratios / labels,
• Optionally give a method extra inputs — register them first with register_attribute / register_noise / register_condition, then list them in order via register_method(..., inputs=[...]); the forward takes them after the audio tensor (forward(self, x, noise, temperature)). These reuse the same role mechanism as neural.gen~,
• Use export_to_pte(..., delegate="coreml") to export, which will result in the model weights .pte and the sidecar .json

import cached_conv as cc
from neural_tilde import LiveModule

cc.use_cached_conv(True)              # MUST be set before building the model

class MyModel(LiveModule):
    def __init__(self):
        super().__init__()
        self.net = cc.CachedSequential(
            cc.Conv1d(1, 16, 3, padding=cc.get_padding(3)),
            cc.Conv1d(16, 1, 3, padding=cc.get_padding(3)),
        )
    def forward(self, x):             # [batch, 1, time] -> [batch, 1, time]
        return self.net(x)

model = MyModel()
model.register_method("forward", in_channels=1, in_ratio=1,
                      out_channels=1, out_ratio=1)
model.export_to_pte("mymodel", delegate="coreml", buffer_size=4096)
# -> mymodel.pte + mymodel.json  (cached_conv state persists across blocks on Core ML)

Example: A complete exemple that builds and exports an untrained conv-net is in examples/export_example.py:

python examples/export_example.py        
# Results in tiny_stream.pte + tiny_stream.json

A model may carry internal mutable state that persists across blocks (what makes a cached_conv net stream click-free); the host loads one instance per object and reuses it, so nothing special is needed in the sidecar. The full producer/consumer protocol is documented in EXECUTORCH_PROTOCOL.md.

If export complains about flatc, prepend <executorch>/cmake-out/third-party/flatc_ep/bin to your PATH.

Generative models for neural.gen~

Offline generators (e.g. text→audio) use neural_tilde.GenModule instead of LiveModule. You wrap an existing nn.Module, declare each input by name with its role — condition (tokens/masks), scalar attribute controls, seeded noise, an optional init buffer — then register_method(...) one or more generation paths, each listing the inputs its method consumes (in forward order) plus the audio output it produces. Just like LiveModule, a model can expose several methods with different inputs (e.g. prompt2audio and audio2audio); the host selects one by name ([neural.gen~ mymodel.pte audio2audio]). The seed and the tokenizer bundle are per-model. It writes a kind:"gen" sidecar .json and, if you register a tokenizer, the *.tokenizer.json + *.tokenizer.config.json bundle that neural.tokenizer loads.

from neural_tilde import GenModule

# methods resolve to my_pipeline.<name>(*inputs), inputs in the registered order:
#   my_pipeline.prompt2audio(input_ids, attention_mask, seconds_total, x, noises) -> [1, 2, length]
#   my_pipeline.audio2audio(init_audio, seconds_total, x, noises)                 -> [1, 2, length]
gm = GenModule(my_pipeline)
gm.register_condition("input_ids", [1, 256], "int64")
gm.register_condition("attention_mask", [1, 256], "int32")
gm.register_attribute("seconds_total", 3.0, 0.0, 384.0,
                      "Total length in seconds to condition the generation on.")
gm.register_noise("x", [1, 256, 32])
gm.register_noise("noises", [7, 1, 256, 32])
gm.register_buffer_input("init_audio", channels=2, length=131072, sample_rate=44100)
gm.register_method("prompt2audio",
                   inputs=["input_ids", "attention_mask", "seconds_total", "x", "noises"],
                   out_channels=2, out_length=131072, out_sample_rate=44100)
gm.register_method("audio2audio",
                   inputs=["init_audio", "seconds_total", "x", "noises"],
                   out_channels=2, out_length=131072, out_sample_rate=44100)
gm.register_tokenizer("tokenizer.json", max_length=256)   # shared by both exporters
gm.set_seed(0)
gm.export_to_pte("mymodel", delegate="mlx", decompose_conv=True)
# -> mymodel.pte + mymodel.json (+ mymodel.tokenizer.json + .tokenizer.config.json)

register_tokenizer(...) lives on the shared exporter base (backed by the neural_tilde.Tokenizer class), so a LiveModule that takes a text prompt — feeding the tokens in as condition inputs — can ship the same standalone tokenizer bundle. The bundle is independent of the model kind.

Example: a tiny stand-in generator (tokens + noise + duration → stereo audio) is in examples/export_gen_example.py:

python examples/export_gen_example.py        # -> tiny_gen.pte + tiny_gen.json

This is the same protocol the Stable Audio 3 export pipeline produces; the full input-role / sidecar schema is §7 of EXECUTORCH_PROTOCOL.md.

Build the externals from source

Please refer to Build.md

Credits

C++ externals and the Python tools derive from nn~ by Antoine Caillon and Axel Chemla--Romeu-Santos (ACIDS-IRCAM). ExecuTorch migration and the neural fork: this repository. License: CC BY-NC 4.0.