mlx-audio-io

mlx-audio-io is the audio data layer for MLX: fast file decode/encode directly to and from mlx.core.array, with one API across macOS and Linux.

Why This Project Exists

MLX has strong tensor and model primitives, but it does not ship a first-class, cross-platform audio file I/O layer comparable to what torchaudio provides in the PyTorch ecosystem.

In practice, MLX users often end up with one of these compromises:

• bridge through NumPy/SoundFile/librosa with extra copies and inconsistent format behavior
• shell out to ffmpeg/ffprobe for non-WAV workflows
• pull in parts of the PyTorch audio stack just to handle common audio containers/codecs

mlx-audio-io closes that gap with a native backend designed for MLX workloads:

• direct decode/encode into mlx.core.array
• one Python API (load, save, info, stream, batch_load, supports_soxr) on both macOS and Linux
• consistent validation and error messages across platforms
• support for training/inference data access patterns (partial reads, chunked streaming, optional resampling)

Platform Backends

• macOS backend optimized for Apple Silicon via AudioToolbox
• Linux backend with native WAV/MP3 fast paths plus libav-backed codec support (FLAC/M4A/AIFF/CAF)

The public Python API is the same on both platforms: load, save, info, stream, batch_load, supports_soxr.

Backend Feature Matrix

Unsupported format/encoding combinations fail with explicit ValueError messages.

Installation

End users (PyPI)

For normal use:

pip install mlx-audio-io

Version policy

MLX and nanobind must be paired

The extension shares nanobind's type registry with mlx.core via NB_DOMAIN, so mlx::core::array crosses the boundary using MLX's own registered caster. Build with a different nanobind than MLX was built with and everything compiles and links cleanly, then every call fails at runtime with Unable to convert function return value to a Python type — an error that mentions nothing about nanobind.

(Taken from MLX's own CMakeLists.txt FetchContent ... GIT_TAG.) You do not normally have to think about this — pip resolves both from [build-system] requires and its default choice is a matching pair. If it ever is not, the build refuses rather than producing a binary that imports and then fails on every call, and it says how to fix it. The build records which nanobind it used, and the loader rejects a mismatched binary at import.

The native extension links libmlx directly, and MLX has no stable C++ ABI — StreamOrDevice gained a variant alternative in MLX 0.32.0, which remangles every operation that takes a stream — so a binary built against 0.31.x cannot load against 0.32.x. Each build records the MLX versions it was compiled and tested against, and the loader rejects anything else at import time rather than crashing later inside a load() or save().

Installing: the extension is compiled on your machine

mlx-audio-io is published as an sdist, not a wheel, so pip install compiles the extension locally. pip does that inside an isolated build environment which it resolves independently of the environment you are installing into — so if you pin an older MLX, pip may still build against a newer one and hand you a binary that cannot load.

If the loader reports a version mismatch, build against your own MLX. A build backend cannot override pip's isolated resolution — pip rejects a conflicting pin outright — so the only way is to supply the build dependencies yourself and turn isolation off:

# nanobind must match your MLX: 0.31.x -> 2.12.0, 0.32.x -> 2.15.0
pip install "mlx==0.31.2" "nanobind==2.12.0" scikit-build-core delocate
pip install --force-reinstall --no-cache-dir --no-build-isolation \
  --no-binary mlx-audio-io mlx-audio-io

(On Linux, swap delocate for auditwheel.) Set MLX_AUDIO_IO_BUILD_MLX to the version you intend to build against and the build will refuse if it is handed a different one, rather than silently producing a binary you cannot load.

Current release:

pip install "mlx-audio-io==1.3.12"

This release builds and runs against:

• macOS: mlx>=0.31.2,<0.33
• Linux: mlx[cpu]>=0.31.2,<0.33

That range is what this source tree can be built against. Which MLX a built extension can load is narrower — exactly the version it was compiled with — and is enforced by the recorded compatible-version list, which fails loudly and names both versions.

To try an unverified MLX version — for a compatibility investigation, not for production — set MLX_AUDIO_IO_ALLOW_MLX_MISMATCH=1, which turns the import-time rejection into a RuntimeWarning. Expect native crashes if the ABI actually differs.

Contributors (source checkout)

For local development and tests:

git clone https://github.com/ssmall256/mlx-audio-io.git
cd mlx-audio-io
uv sync --extra dev

Hard Rule: Do Not Copy .venv Between Machines

Do not copy project virtual environments across machines. Native extensions can fail integrity/code-sign checks or crash when moved between hosts.

If you already copied one, recreate it:

rm -rf .venv && uv venv --python 3.11 && uv sync

Linux source build behavior

Linux source builds require libav and use direct libav-backed paths:

• Linux info() for non-WAV formats uses direct libav metadata.
• Linux load() for non-WAV formats uses direct libav decode for all offset/duration combinations.
• Linux stream() for non-WAV formats uses direct libav packet/frame decode.
• Linux save() for encoded formats (.mp3, .flac, .m4a, .aiff/.aif, .caf) uses direct libav encode/mux.

Wheel portability and libsoxr

• soxr_hq / soxr_vhq are optional and enabled only when libsoxr is available at build time.
• macOS wheel builds are automatically repaired with delocate in this project build backend, so external libsoxr dylibs are bundled into the wheel instead of relying on Homebrew paths.
• Linux wheel repair via auditwheel is supported and can be enabled with MLX_AUDIO_IO_REPAIR_LINUX=1 in release CI.
• To disable wheel repair explicitly (not recommended for release builds), set MLX_AUDIO_IO_REPAIR_WHEEL=0.
• Release CI verifies wheel linkage with tools/check_wheel_linkage.py and fails if absolute host library paths are detected.
• Third-party notices for bundled libsoxr are shipped in-package at:
  • mlx_audio_io/THIRD_PARTY_NOTICES.md
  • mlx_audio_io/licenses/libsoxr/
• Current release publish workflows build and publish sdist artifacts; a separate macOS wheel job verifies wheel linkage and notice packaging as a release gate.

Requirements

• Python 3.10+
• Runtime:
  • macOS: Apple Silicon + mlx
  • Linux: mlx[cpu] (current default)
• Source builds:
  • CMake 3.24+, C++20 toolchain, pkg-config
  • Linux default build: libavformat-dev, libavcodec-dev, libavutil-dev, libswresample-dev

Troubleshooting

• ModuleNotFoundError: mlx_audio_io
  • Install in the project environment (uv sync) and run via uv run ....
• ImportError for mlx on Linux
  • Ensure Linux dependency is installed as mlx[cpu].
• Build failures on source installs
  • Verify build-essential, cmake, ninja-build, and pkg-config are installed.
• Extended Linux format support errors (.mp3, .m4a, .flac, .aiff, .caf)
  • For default Linux builds, ensure runtime libav libraries are present (libavformat, libavcodec, libavutil, libswresample).
• MP3 test fixture generation failures
  • Tests that generate MP3 fixtures require ffmpeg or lame available on PATH.
• Native import failures or unexpected crashes
  • Run diagnostics: python -m mlx_audio_io.doctor
  • Check MLX runtime compatibility: python -c "import mlx_audio_io as aio; print(aio.show_build_info())"
  • If build_mlx_version and runtime_mlx_version differ, rebuild against the MLX you run — see "Installing: the extension is compiled on your machine". The error the loader prints contains the exact two commands.
  • Moving your runtime instead also works, if nothing else pins it: pip install -U "mlx==<build_mlx_version>"
  • Avoid pip install --no-deps for mlx-audio-io unless you manually pin a matching mlx version.
  • Recreate env (do not copy .venv between machines): rm -rf .venv && uv venv --python 3.11 && uv sync

Quickstart

from mlx_audio_io import load, save, info, stream, batch_load, supports_soxr

# Load
x, sr = load("speech.wav")

# Resample + mono (auto-selects soxr_vhq when available, falls back to "best")
x16, sr16 = load("speech.wav", sr=16000, mono=True)

# Metadata without decoding
meta = info("speech.wav")

# Stream in chunks
for chunk, chunk_sr in stream("long.wav", chunk_duration=2.0):
    pass

# Save WAV
save("out.wav", x, sr)
save("out_pcm16.wav", x, sr, encoding="pcm16")

# Batch load
items = batch_load(["a.wav", "b.wav"], sr=16000, mono=True)

Additional save examples:

save("out.flac", x, sr)
save("out.mp3", x, sr, bitrate="192k")
save("out.m4a", x, sr, bitrate="256k")
save("out.m4a", x, sr, encoding="alac")

API Reference

load

load(path, sr=None, offset=0.0, duration=None, mono=False, mono_mode="mean",
     layout="channels_last", dtype="float32", resample_quality="default")

Decode audio into an mlx.core.array. Returns (audio, sample_rate).

On Linux WAV/MP3 fast paths, resample quality levels currently map to the same linear behavior. soxr_hq/soxr_vhq use true libsoxr resampling (when built with libsoxr). If soxr_hq/soxr_vhq is requested without libsoxr support, load()/resample() raise RuntimeError. torchaudio_compat requires torch + torchaudio and uses torchaudio.functional.resample.

When sr is specified and resample_quality is left at "default", load() automatically selects soxr_vhq when libsoxr is available, falling back to "best" otherwise. You can still override explicitly:

audio, sr = load("speech.wav", sr=16000, resample_quality="soxr_hq")

batch_load

batch_load(paths, sr=None, mono=False, mono_mode="mean", dtype="float32", num_workers=4)

Threaded multi-file load(). Returns list[(audio, sample_rate)].

save

save(path, audio, sr, layout="channels_last", encoding="float32",
     bitrate="auto", clip=True)

Write audio from mx.array (or numpy.ndarray) to disk.

stream

stream(path, chunk_frames=None, chunk_duration=None, sr=None,
       mono=False, mono_mode="mean", dtype="float32", offset=0.0, duration=None)

Return an iterator yielding (audio_chunk, sample_rate). Exactly one of chunk_frames or chunk_duration is required.

info

info(path)

Return AudioInfo metadata without decoding sample buffers.

supports_soxr

supports_soxr()

Return True when the installed native extension was built with libsoxr support. Use this to select resample_quality at runtime.

Testing

Run all tests:

uv sync --extra dev
uv run python -m pytest -q

Run Linux supported subset:

uv run python -m pytest -q -m "not apple_only"

Run Apple-only subset:

uv run python -m pytest -q -m "apple_only"

Linux Docker run from a macOS host:

docker run --rm -it --platform linux/arm64 \
  -v "$PWD":/work -w /work \
  python:3.14-bookworm bash -lc '
    apt-get update && apt-get install -y --no-install-recommends \
      build-essential cmake ninja-build pkg-config ffmpeg \
      libavformat-dev libavcodec-dev libavutil-dev libswresample-dev &&
    python -m pip install -U pip uv &&
    uv sync --extra dev &&
    uv run python -m pytest -q -m "not apple_only"
  '

Performance

Benchmark methodology, commands, and full result tables live in docs/benchmarking.md.

Headline numbers (194.8s stereo PCM16 WAV @ 44.1 kHz, p50 median latency):

Full tables with torchaudio comparisons, M1 Max, and Linux x86_64 results are in the benchmarking doc.

License

MIT