torch-dae: an AI skill-based framework for Audio Embedding Models

torch-dae is a reproducible integration and profiling framework for PyTorch audio models. It connects one explicit model variant and checkpoint to a validated runtime environment, a typed waveform/output interface, runtime-verification evidence, and optional performance Technical Cards. The repository also ships agent skills for controlled model onboarding and profiling.

What you can do

• Use accepted audio models through one typed PyTorch wrapper contract.
• Extract checkpoint-specific embeddings with declared dimensions and semantics.
• Recreate model runtimes reproducibly from committed environment definitions instead of installing model dependencies into the root package.
• Acquire checkpoints explicitly through a content-addressed cache; weights are never silently downloaded during import or wrapper construction.
• Inspect checkpoint-specific Model Cards describing input, output, embedding, runtime, and limitation contracts.
• Profile accepted models on CPU, MPS, and CUDA selectors supported by the local machine and generate candidate Technical Cards with raw timing measurements.
• Integrate additional models with the audio-model-onboarding skill through controlled analysis, environment resolution, integration, verification, and card-generation phases.

The framework deliberately separates model execution from repository control. The root package stays lightweight; heavyweight or mutually incompatible model dependencies live in isolated, model-specific environments.

Supported models

The repository currently contains three accepted PANNs / AudioSet checkpoint integrations. Each is runtime_verified and has canonical Profiling v1 Technical Cards for CPU and Apple MPS execution contexts.

CUDA is declared by the upstream PANNs implementation but has not been locally verified by the accepted Model Cards. The currently accepted PANNs environment definitions constrain managed materialization to macOS on arm64. See PANNs integration for the exact runtime and duration constraints.

Installation

Package installation

The published distribution is named torch-deepaudioembedding; the import package is torch_dae and the console command is torch-dae.

pip install torch-deepaudioembedding
python -c "import torch_dae"
torch-dae --help

The wheel provides the Python package and CLI control plane. It intentionally does not bundle the repository's Model Cards, environment definitions, verification reports, Technical Cards, agent skills, or model checkpoints.

Full repository workspace

Use a source checkout for the complete registry-backed workflow: supported models, managed environments, checkpoint acquisition, runtime verification, profiling, and the agent skills.

git clone https://github.com/StefanoGiacomelli/torch_dae.git
cd torch_dae
uv sync --python 3.11 --all-groups --extra profiling --frozen
uv run torch-dae --help

Model-specific PyTorch stacks are not installed into this root environment. They are materialized separately from environments/<environment-id>/ when a model runtime is needed.

Full installation guidance: docs/getting-started/installation.md.

Quick start

From a source checkout:

uv run torch-dae card list
uv run torch-dae card show panns-cnn14-16k-map-0438
uv run torch-dae technical-card list

Expected Model Card IDs:

panns-cnn14-16k-map-0438
panns-resnet38-map-0434
panns-wavegram-logmel-cnn14-map-0439

The first command discovers accepted model identities without importing the model runtime. The second prints the normalized checkpoint-specific Model Card. The third lists canonical empirical profiling evidence already accepted under technical_cards/.

For an end-to-end pretrained PANNs example, including environment creation, checkpoint acquisition, and Python inference, follow the PANNs inference tutorial.

Model execution at a glance

A supported wrapper receives a floating-point waveform tensor and an integer native sample rate. For the current PANNs integrations, the public input layout is always:

[B, 1, T]

where B is batch size, the channel dimension must be exactly 1, and T is the number of audio samples. The wrappers do not automatically resample, downmix, normalize amplitude, pad, crop, or truncate the input.

Inside the verified model environment, the core API is:

from pathlib import Path

import torch

from torch_dae import ModelCardRegistry

card_id = "panns-cnn14-16k-map-0438"
checkpoint_path = Path("/absolute/path/to/Cnn14_16k_mAP=0.438.pth")

registry = ModelCardRegistry(Path.cwd())
model_class = registry.get_model_class(card_id)
model = model_class.from_pretrained(checkpoint_path).eval()

waveform = torch.zeros(1, 1, 16_000, dtype=torch.float32)

with torch.no_grad():
    output = model(waveform, 16_000)
    embedding = model.compute_embedding(waveform, 16_000)

print(output.primary.shape)                  # torch.Size([1, 527])
print(output.tensors["probabilities"].shape) # torch.Size([1, 527])
print(embedding.tensor.shape)                # torch.Size([1, 2048])

from_pretrained() never downloads a default checkpoint. Supply a checkpoint already materialized by torch-dae checkpoint ensure <card-id> or another explicitly controlled path.

PANNs output contract

Do not apply a softmax to the 527-class output: PANNs performs multi-label audio tagging and exposes independent sigmoid probabilities. No threshold or class aggregation is applied by the wrapper.

Detailed execution semantics: docs/user-guide/model-execution.md.

Embeddings

All three current PANNs Model Cards expose one verified default embedding:

panns-official-post-fc1-embedding

It is a clipwise [B,2048] representation produced after ReLU(fc1) and the following dropout call, immediately before the final AudioSet classifier. In evaluation mode the dropout call is the identity.

embedding = model.compute_embedding(
    waveform,
    sample_rate,
    embedding_id=None,  # use the Model Card default
)
print(embedding.embedding_id)
print(embedding.layout)       # B,D
print(embedding.tensor.shape) # [B,2048]

See docs/user-guide/embeddings.md.

Profiling a supported model

Profiling is optional empirical evidence and never mutates the accepted Model Card. A profiling run creates candidate Technical Card JSON plus compact raw .npz measurements in a repository-local candidate directory.

Example CPU run:

uv run torch-dae model profile \
  --model panns-cnn14-16k-map-0438 \
  --device cpu \
  --energy off \
  --output-dir profiling_candidates/panns-cnn14-cpu

The user-selectable controls are:

--model
--device auto|cpu|mps|cuda|cuda:<index>   (repeatable)
--protocol audio-inference-v1
--energy auto|off
--allow-privileged-energy
--output-dir
--json

Profiling v1 fixes the comparable measurement protocol rather than exposing every benchmark detail as a CLI knob: deterministic seeded float32 white noise, batch sizes 1,2,4,8, 10 warmups, 50 measured steady-state inferences, and both single_thread and native_default regimes for CPU.

After a run, inspect the Technical Card JSON for summarized architecture, latency, throughput, real-time factor, memory, energy coverage, and minimum-input evidence. Use the accompanying .npz when you need the raw latency observations. Canonical Technical Cards already accepted for the PANNs models live under technical_cards/<model-id>/.

Profiling guide: docs/profiling/overview.md.

Model Cards and Technical Cards

These artifacts answer different questions:

One checkpoint-specific Model Card may therefore have zero, one, or many Technical Cards. Profiling adds evidence; it does not change model identity or acceptance status.

Integrating a new model with the skill

The canonical audio-model-onboarding skill separates integration into reviewable phases:

analyze
  ↓
resolve-environment
  ↓
integrate
  ↓
verify
  ↓
card

Run one phase per agent request and review the produced evidence before authorizing the next phase. A good first request is:

Use the canonical `audio-model-onboarding` skill available in this repository.

MODE: analyze
WORKFLOW_ID: <STABLE_WORKFLOW_ID_OR_AUTO_DISCOVER>

MODEL_NAME: <MODEL_NAME>
UPSTREAM_REPOSITORY: <GITHUB_REPOSITORY_URL>
PAPER_OR_TECHNICAL_REFERENCE: <PAPER_URL_OR_NONE>

TARGET_VARIANT: <VARIANT_NAME_OR_AUTO_DISCOVER>
TARGET_CHECKPOINT: <CHECKPOINT_NAME_OR_AUTO_DISCOVER>
PREFERRED_EMBEDDING: <EMBEDDING_NAME_OR_UNRESOLVED>

ADDITIONAL_CONSTRAINTS:
<OPTIONAL_PROJECT_SPECIFIC_CONDITIONING_OR_NONE>

Inspect the upstream project statically, identify the supported model/checkpoint/embedding
candidates, report unresolved decisions, and stop at the analyze-mode boundary. Do not continue to
environment resolution until I explicitly approve the analysis.

The exact canonical template is in skills/audio-model-onboarding/templates/agent-request.md. The onboarding skill and the independent profiling skill are documented under skills/ and in the documentation. A larger copy-paste prompt library is part of the user documentation work for the next release.

Repository layout

src/torch_dae/                         Python control plane and public wrappers
model_cards/                           Accepted checkpoint-specific Model Cards
environments/                          Committed isolated-runtime definitions
verification_reports/                  Accepted checkpoint runtime observations
technical_cards/                       Accepted empirical profiling evidence
onboarding_reports/                    Accepted cross-phase onboarding artifacts
skills/audio-model-onboarding/         Canonical model-integration agent skill
skills/audio-model-profiling/          Canonical profiling agent skill
schemas/                               Generated strict JSON Schemas
docs/                                  User, API, workflow, and developer documentation
tests/                                 Contract, runtime, safety, and regression tests
.torch-dae/                            Ignored local runtime/cache/workspace state

Model checkpoint payloads are not committed to the repository.

Documentation

Read the hosted documentation at https://torch-dae.readthedocs.io/en/stable/.

Recommended entry points:

• Installation
• Mental model
• Quickstart
• Supported models
• PANNs inference tutorial
• Model execution
• Embeddings
• Profiling
• Onboarding skill
• Python API

Development

The root development environment remains model-agnostic. Model runtimes are exercised through their isolated environments rather than by adding heavyweight dependencies to the package root.

uv sync --python 3.11 --all-groups --extra profiling --frozen
uv run --python 3.11 --all-groups --extra profiling --frozen ruff format --check
uv run --python 3.11 --all-groups --extra profiling --frozen ruff check
uv run --python 3.11 --all-groups --extra profiling --frozen mypy src scripts
uv run --python 3.11 --all-groups --extra profiling --frozen pytest
uv run --python 3.11 --all-groups --extra profiling --frozen python scripts/validate_repository.py
uv run --python 3.11 --all-groups --extra profiling --frozen sphinx-build -W --keep-going -b html docs docs/_build/html

See CONTRIBUTING.md for the full contribution contract.

Current boundaries

• The repository does not redistribute pretrained checkpoint payloads; accepted assets are acquired explicitly from their authoritative providers.
• The current accepted PANNs managed environments are constrained to macOS/arm64. CUDA support is upstream-declared but not locally verified by the accepted cards.
• PANNs inputs must already be mono floating-point tensors at the exact native sample rate. Automatic resampling, downmixing, amplitude normalization, padding, cropping, and truncation are not part of the wrapper contract.
• Padded batches with unequal valid_lengths are not currently supported by the PANNs wrappers.
• torch-dae model inspect remains an explicit unavailable-feature placeholder.
• Candidate profiling evidence is never promoted automatically into technical_cards/; promotion is a separate human-reviewed repository operation.

Funding

The research project title is:

Methods of Computational Auditory Scene Analysis and Synthesis supporting eXtended and Immersive Reality Services

Research activities were mainly funded under the Ministerial Decree (DM) 118/2023, Mission 4, Component 1, Investment 4.1 of the National Recovery and Resilience Plan (PNRR) – “PNRR Research” – CUP: E11I23000100001.

Citations

Use the repository's CITATION.cff for software citation metadata. The stable concept DOI for the complete software series is 10.5281/zenodo.21641390. Zenodo assigns an immutable version-specific DOI after each GitHub release is archived; use that record when citing an exact release.

When discussing the framework design, standardization rationale, or deployment methodology, also cite:

@inproceedings{giacomelli2025torch_dae,
  author    = {Giacomelli, Stefano and Centofanti, Carlo and
               Graziosi, Fabio and Rinaldi, Claudia},
  title     = {{Design and Deployment of a Standard Framework for
                Audio Neural Networks Embedding Models}},
  booktitle = {2025 IEEE Symposium on Computers and Communications (ISCC)},
  year      = {2025},
  pages     = {1--6},
  doi       = {10.1109/ISCC65549.2025.11326439},
  publisher = {IEEE}
}

License

Licensed under the Apache License 2.0. Attribution information is provided in NOTICE.

Contact

Stefano Giacomelli
ICT - Ph.D. Candidate
Department of Information Engineering, Computer Science and Mathematics (DISIM)
University of L'Aquila, Italy

• Email: stefano.giacomelli@graduate.univaq.it
• GitHub: StefanoGiacomelli
• ORCID: 0009-0009-0438-1748
• Google Scholar: Profile
• LinkedIn: Profile