mlx-audio-io

mlx-audio-io is the audio data layer for MLX: fast file decode/encode directly to and from mlx.core.array, with one API across macOS and Linux.

Why This Project Exists

MLX has strong tensor and model primitives, but it does not ship a first-class, cross-platform audio file I/O layer comparable to what torchaudio provides in the PyTorch ecosystem.

In practice, MLX users often end up with one of these compromises:

• bridge through NumPy/SoundFile/librosa with extra copies and inconsistent format behavior
• shell out to ffmpeg/ffprobe for non-WAV workflows
• pull in parts of the PyTorch audio stack just to handle common audio containers/codecs

mlx-audio-io closes that gap with a native backend designed for MLX workloads:

• direct decode/encode into mlx.core.array
• one Python API (load, save, info, stream, batch_load, supports_soxr) on both macOS and Linux
• consistent validation and error messages across platforms
• support for training/inference data access patterns (partial reads, chunked streaming, optional resampling)

Platform Backends

• macOS backend optimized for Apple Silicon via AudioToolbox
• Linux backend with native WAV/MP3 fast paths plus libav-backed codec support (FLAC/M4A/AIFF/CAF)

The public Python API is the same on both platforms: load, save, info, stream, batch_load, supports_soxr.

Backend Feature Matrix

Unsupported format/encoding combinations fail with explicit ValueError messages.

Installation

End users (PyPI)

For normal use:

pip install mlx-audio-io

Version policy

MLX and nanobind must be paired

The extension shares nanobind's type registry with mlx.core via NB_DOMAIN, so mlx::core::array crosses the boundary using MLX's own registered caster. Build with a different nanobind than MLX was built with and everything compiles and links cleanly, then every call fails at runtime with Unable to convert function return value to a Python type — an error that mentions nothing about nanobind.

(Taken from MLX's own CMakeLists.txt FetchContent ... GIT_TAG.) The build backend selects nanobind after MLX is known. For a regular isolated pip build it requests a verified default pair; uv can supply its locked runtime MLX to the build environment first. The build records both versions and rejects a mismatched pair before compiling.

The native extension links libmlx directly, and MLX has no stable C++ ABI — StreamOrDevice gained a variant alternative in MLX 0.32.0, which remangles every operation that takes a stream — so a binary built against 0.31.x cannot load against 0.32.x. Each build records the MLX versions it was compiled and tested against, and the loader rejects anything else at import time rather than crashing later inside a load() or save().

Installing: the extension is compiled on your machine

mlx-audio-io is published as an sdist, not a wheel, so pip install compiles the extension locally. pip does that inside an isolated build environment which it resolves independently of the environment you are installing into — so if you pin an older MLX, pip may still build against a newer one and hand you a binary that cannot load.

For uv projects, add this to the consuming project's pyproject.toml and update its lockfile once. uv sync --frozen will then build this package using the exact MLX version selected for the project, including editable installs:

[tool.uv.extra-build-dependencies]
mlx-audio-io = [{ requirement = "mlx[cpu]", match-runtime = true }]

Run uv lock after adding the setting, then use uv sync --frozen on each machine.

This repository uses the same setting for its own editable build. The build hook then supplies the matching nanobind automatically. If MLX is upgraded in the lockfile, uv rebuilds the extension for that version. The setting must be in each consumer's project configuration; a library cannot impose it on an unrelated uv project. The cpu extra installs mlx-cpu on Linux and has no additional dependency on macOS.

For pip, isolated builds still select build MLX independently of runtime MLX. If the loader reports a version mismatch, rebuild against your runtime MLX by supplying build dependencies and turning isolation off:

# nanobind must match your MLX; MLX 0.32.3 uses nanobind 3.0.1
pip install "mlx==0.32.3" "nanobind==3.0.1" scikit-build-core cmake ninja delocate
pip install --force-reinstall --no-cache-dir --no-build-isolation \
  --no-binary mlx-audio-io --no-deps mlx-audio-io

cmake and ninja are in that list because --no-build-isolation means pip installs nothing for the build: scikit-build-core normally declares them itself, and turning isolation off skips that. The rebuild takes about ten seconds once the tooling is present.

(On Linux, swap delocate for auditwheel.) Set MLX_AUDIO_IO_BUILD_MLX to the version you intend to build against and the build will refuse if it is handed a different one, rather than silently producing a binary you cannot load.

Install:

pip install mlx-audio-io

It builds and runs against:

• macOS: mlx>=0.31.2,<0.33
• Linux: mlx[cpu]>=0.31.2,<0.33

That range is what this source tree can be built against. Which MLX a built extension can load is narrower — exactly the version it was compiled with — and is enforced by the recorded compatible-version list, which fails loudly and names both versions.

To try an unverified MLX version — for a compatibility investigation, not for production — set MLX_AUDIO_IO_ALLOW_MLX_MISMATCH=1, which turns the import-time rejection into a RuntimeWarning. Expect native crashes if the ABI actually differs.

Contributors (source checkout)

For local development and tests:

git clone https://github.com/ssmall256/mlx-audio-io.git
cd mlx-audio-io
uv sync --extra dev

To create a standalone sdist or wheel with uv build, pass --no-config. The match-runtime setting above applies to environment sync; an artifact build has no runtime lock to match.

Hard Rule: Do Not Copy .venv Between Machines

Do not copy project virtual environments across machines. Native extensions can fail integrity/code-sign checks or crash when moved between hosts.

If you already copied one, recreate it:

rm -rf .venv && uv venv --python 3.11 && uv sync

Linux source build behavior

Linux source builds require libav and use direct libav-backed paths:

• Linux info() for non-WAV formats uses direct libav metadata.
• Linux load() for non-WAV formats uses direct libav decode for all offset/duration combinations.
• Linux stream() for non-WAV formats uses direct libav packet/frame decode.
• Linux save() for encoded formats (.mp3, .flac, .m4a, .aiff/.aif, .caf) uses direct libav encode/mux.

Wheel portability and libsoxr

• soxr_hq / soxr_vhq are optional and enabled only when libsoxr is available at build time.
• macOS wheel builds are automatically repaired with delocate in this project build backend, so external libsoxr dylibs are bundled into the wheel instead of relying on Homebrew paths.
• Linux wheel repair via auditwheel is supported and can be enabled with MLX_AUDIO_IO_REPAIR_LINUX=1 in release CI.
• To disable wheel repair explicitly (not recommended for release builds), set MLX_AUDIO_IO_REPAIR_WHEEL=0.
• Release CI verifies wheel linkage with tools/check_wheel_linkage.py and fails if absolute host library paths are detected.
• Third-party notices for bundled libsoxr are shipped in-package at:
  • mlx_audio_io/THIRD_PARTY_NOTICES.md
  • mlx_audio_io/licenses/libsoxr/
• Current release publish workflows build and publish sdist artifacts; a separate macOS wheel job verifies wheel linkage and notice packaging as a release gate.

Requirements

• Python 3.10+
• Runtime:
  • macOS: Apple Silicon + mlx
  • Linux: mlx[cpu] (current default)
• Source builds:
  • CMake 3.24+, C++20 toolchain, pkg-config
  • Linux default build: libavformat-dev, libavcodec-dev, libavutil-dev, libswresample-dev

Troubleshooting

• ModuleNotFoundError: mlx_audio_io
  • Install in the project environment (uv sync) and run via uv run ....
• ImportError for mlx on Linux
  • Ensure Linux dependency is installed as mlx[cpu].
• Build failures on source installs
  • Verify build-essential, cmake, ninja-build, and pkg-config are installed.
• Extended Linux format support errors (.mp3, .m4a, .flac, .aiff, .caf)
  • For default Linux builds, ensure runtime libav libraries are present (libavformat, libavcodec, libavutil, libswresample).
• MP3 test fixture generation failures
  • Tests that generate MP3 fixtures require ffmpeg or lame available on PATH.
• Native import failures or unexpected crashes
  • Run diagnostics: python -m mlx_audio_io.doctor
  • Check MLX runtime compatibility: python -c "import mlx_audio_io as aio; print(aio.show_build_info())"
  • If build_mlx_version and runtime_mlx_version differ, rebuild against the MLX you run — see "Installing: the extension is compiled on your machine". The error the loader prints contains the exact two commands.
  • Moving your runtime instead also works, if nothing else pins it: pip install -U "mlx==<build_mlx_version>"
  • Avoid pip install --no-deps for mlx-audio-io unless you manually pin a matching mlx version.
  • Recreate env (do not copy .venv between machines): rm -rf .venv && uv venv --python 3.11 && uv sync

Quickstart

from mlx_audio_io import load, save, info, stream, batch_load, supports_soxr

# Load
x, sr = load("speech.wav")

# Resample + mono (auto-selects soxr_vhq when available, falls back to "best")
x16, sr16 = load("speech.wav", sr=16000, mono=True)

# Metadata without decoding
meta = info("speech.wav")

# Stream in chunks
for chunk, chunk_sr in stream("long.wav", chunk_duration=2.0):
    pass

# Save WAV
save("out.wav", x, sr)
save("out_pcm16.wav", x, sr, encoding="pcm16")

# Batch load
items = batch_load(["a.wav", "b.wav"], sr=16000, mono=True)

Additional save examples:

save("out.flac", x, sr)
save("out.mp3", x, sr, bitrate="192k")
save("out.m4a", x, sr, bitrate="256k")
save("out.m4a", x, sr, encoding="alac")

API Reference

load

load(path, sr=None, offset=0.0, duration=None, mono=False, mono_mode="mean",
     layout="channels_last", dtype="float32", resample_quality="default",
     low_memory=False, squeeze_mono=False)

Decode audio into an mlx.core.array. Returns (audio, sample_rate).

On macOS, default resampling when sr is specified automatically routes through Apple AudioToolbox (best), delivering ~4.5x faster throughput than soxr_vhq with >85 dB SNR. Set MLX_AUDIO_IO_SOXR_DEFAULT=1 to use libsoxr instead. On Linux WAV/MP3 fast paths, resample quality levels currently map to the same linear behavior. soxr_hq/soxr_vhq use true libsoxr resampling (when built with libsoxr). If soxr_hq/soxr_vhq is requested without libsoxr support, load()/resample() raise RuntimeError. torchaudio_compat requires torch + torchaudio and uses torchaudio.functional.resample.

When sr is specified and resample_quality is left at "default", load() automatically selects best on macOS and soxr_vhq on Linux when libsoxr is available. You can still override explicitly:

audio, sr = load("speech.wav", sr=16000, resample_quality="soxr_hq")

batch_load

batch_load(paths, sr=None, mono=False, mono_mode="mean", dtype="float32", num_workers=4, squeeze_mono=False)

Threaded multi-file load(). Returns list[(audio, sample_rate)].

save

save(path, audio, sr, layout="auto", encoding="float32",
     bitrate="auto", clip=True)

Write audio from mx.array (or numpy.ndarray) to disk.

stream

stream(path, chunk_frames=None, chunk_duration=None, sr=None,
       mono=False, mono_mode="mean", dtype="float32", offset=0.0, duration=None)

Return an iterator yielding (audio_chunk, sample_rate). Exactly one of chunk_frames or chunk_duration is required.

info

info(path)

Return AudioInfo metadata without decoding sample buffers.

supports_soxr

supports_soxr()

Return True when the installed native extension was built with libsoxr support. Use this to select resample_quality at runtime.

Testing

Run all tests:

uv sync --extra dev
uv run python -m pytest -q

Run Linux supported subset:

uv run python -m pytest -q -m "not apple_only"

Run Apple-only subset:

uv run python -m pytest -q -m "apple_only"

Linux Docker run from a macOS host:

docker run --rm -it --platform linux/arm64 \
  -v "$PWD":/work -w /work \
  python:3.14-bookworm bash -lc '
    apt-get update && apt-get install -y --no-install-recommends \
      build-essential cmake ninja-build pkg-config ffmpeg \
      libavformat-dev libavcodec-dev libavutil-dev libswresample-dev &&
    python -m pip install -U pip uv &&
    uv sync --extra dev &&
    uv run python -m pytest -q -m "not apple_only"
  '

Performance

Benchmark methodology, commands, and full result tables live in docs/benchmarking.md.

Headline numbers (194.8s stereo PCM16 WAV @ 44.1 kHz, p50 median latency):

Full tables with torchaudio comparisons, M1 Max, and Linux x86_64 results are in the benchmarking doc.

License

MIT