SONAR-OSS is Poseidon's open-source toolkit for evaluating automatic speech recognition (ASR) models — metrics, reporting, and benchmarks, distributed as the psdn-sonar Python package. Point it at your audio and reference transcripts, pick models (local HuggingFace checkpoints or hosted APIs like OpenAI Whisper, ElevenLabs, and AssemblyAI), and it produces per-sample and aggregate scores you can actually compare: WER/CER under language-specific normalization contracts, semantic similarity, the composite POSEIDON score, audio-quality diagnostics, and latency — with every run's model revision, device, and normalization recorded in the artifact so the numbers stay reproducible.

Where SONAR-OSS earns its keep is off the beaten benchmark path:

• Low-resource languages first. Bengali, Hindi, and Korean ship with curated model defaults and dedicated normalizers alongside English; any other language runs through a YAML recipe (psdn-sonar custom) — adding a language means writing config, not code.
• Single- and multi-speaker evaluation. The multi-speaker pipeline handles VAD, diarization, and channel-split preprocessing (pyannote), and scores the pipeline end to end.
• Scores you can defend. Artifacts carry lineage (checkpoint SHA, normalization contract, device) and machine-readable warnings; nothing is back-solved or invented. psdn-sonar leaderboard renders a comparison table from measured runs only.
• One CLI for the whole loop. discover prepares datasets, single / multi / custom evaluate, leaderboard compares — CSV, JSON, Markdown, and plots out.

If SONAR-OSS is useful to you, please ⭐ star the repo — it helps other voice-AI teams find it and tells us where to invest.

Status: pre-release. The library (metrics, language processors, dataset loaders, evaluators, reporting, CLI) is in place and heading toward a first 0.1.0 release; see CHANGELOG.md. Content imported from the upstream codebase passes the checklist in docs/import-gate.md.

Table of contents

• Requirements
• Installation
  • Install from PyPI
  • Contributor install (from source)
  • API keys and gated models
• Usage
• Development
• Community and links
• License

Requirements

• Python 3.10–3.12
• uv (recommended) or pip
• ffmpeg required by the pipeline-based ASR adapters — including the English defaults whisper_base_en / whisper_small_en and any --hf-model that falls back to the generic pipeline — for all audio input, WAV included (the package decodes every input file with ffmpeg itself and hands the pipeline a raw array, so one decoder covers WAV, M4A/AAC and everything else). These adapters refuse to load without it and name the missing binary. Adapters that decode through libsndfile (the wav2vec2_* models and the non-pipeline Whisper fine-tunes) evaluate WAV and FLAC — and MP3, with the libsndfile ≥ 1.1 that current soundfile wheels bundle — without ffmpeg; formats libsndfile cannot read (M4A/AAC) and pydub silence-trimming of non-WAV input still need it. Install: sudo apt-get install ffmpeg (Debian/Ubuntu), brew install ffmpeg (macOS), or winget install Gyan.FFmpeg (Windows). The [pyannote] extra needs ffmpeg too, and needs more than the binary: pyannote.audio 4.x decodes audio through torchcodec, which loads the ffmpeg shared libraries at runtime. On Windows that means the full-shared build — winget install Gyan.FFmpeg.Shared — because the default Gyan.FFmpeg build ships only ffmpeg.exe and no DLLs; the package registers the DLL-carrying ffmpeg directories for resolution itself (Python ≥ 3.8 does not consult PATH for a native extension's dependent DLLs). Both builds can be installed side by side in either PATH order: every ffmpeg directory on PATH is inspected and the ones holding the shared libraries are registered, so the static build shadowing the shared one does not matter. Linux and macOS package managers install the shared libraries alongside the binary, so no extra step is needed there.

Supported environments. CI validates Linux x86_64 with CPython 3.10, 3.11, and 3.12, plus macOS arm64 with CPython 3.12 and the [ml] extra installed (so the HuggingFace model adapters are exercised on every PR, not just importable). Windows is expected to work for the core package but is not CI-validated; some optional extras have platform-sensitive dependencies ([korean] needs a Java runtime at runtime, [ml]/[pyannote] pull large PyTorch trees). Open an issue if an install fails on a supported Python. The checked-in .python-version pins uv to CPython 3.12, so make setup on a machine with no Python selected builds a supported interpreter instead of whatever newest version uv manages (CI passes --python explicitly and is unaffected).

What a first run costs. On a fresh machine, budget several gigabytes of downloads and tens of minutes before the first number appears; everything is cached, so later runs against the same data and models start in seconds. Measured example (Bengali FLEURS + one CTC model): ~3.4 GB / ~27 min for the dataset, 1.26 GB for the model checkpoint, and ~1.5 GB on disk for the [ml] extra (torch alone ~0.5 GB). psdn-sonar discover --max-samples bounds how many samples are prepared, not the download — each requested split is fetched into the HuggingFace cache in full on first run. Two more downloads happen lazily: the first POSEIDON / semantic-similarity call fetches the ~64 MB sentence-transformers scorer, and audio-quality analysis fetches a ~390 MB UTMOS checkpoint via torch.hub. Checkpoint size can also hide behind an adapter: khushids_bengali is a 62 MB PEFT adapter whose openai/whisper-large-v3 base adds ~2.9 GB.

Compute device and runtime. Local HuggingFace adapters auto-select the best available device — CUDA, then MPS (Apple Silicon), then CPU — and the device used is recorded in each run's scores_<model>.json (submission.device), since a GPU run and a CPU run can produce different transcripts for the same audio. On CPU, architecture dominates runtime: measured on the same 200-utterance FLEURS Bengali set, a Wav2Vec2 CTC model ran at ~1.2 s/sample while a Whisper-medium fine-tune ran at ~19 s/sample. A full multi-model language default (Bengali has 9 models, mostly Whisper-class) is a multi-hour job without a GPU — trim --models and --max-samples to size your run first.

Installation

Install from PyPI

Requires Python 3.10, 3.11, or 3.12.

1. Create and activate a fresh virtual environment. These examples use Python 3.12; substitute 3.10 or 3.11 if needed.

   macOS or Linux (bash/zsh):

   python3.12 --version
   python3.12 -m venv sonar-env
   source sonar-env/bin/activate
   

   Windows PowerShell:

   py -3.12 --version
   py -3.12 -m venv sonar-env
   Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope Process
   .\sonar-env\Scripts\Activate.ps1
   python -c "import sys; print(sys.executable)"   # must print ...\sonar-env\Scripts\python.exe
   

   The Set-ExecutionPolicy line matters: Windows client editions default to the Restricted policy, which refuses Activate.ps1 with an error on stderr while the shell continues with exit 0 — so without it, python still points at the previous interpreter and step 2 silently installs outside the environment. -Scope Process changes the policy for this PowerShell session only. The sys.executable check confirms activation actually took. From cmd.exe, use sonar-env\Scripts\activate.bat instead, which needs no policy change.

2. Install the package:

   python -m pip install "psdn-sonar>=0.1.1"
   

   To install an optional extra, name it in brackets — for example python -m pip install "psdn-sonar[ml]>=0.1.1" for the local-model backends. CONTRIBUTING.md lists every extra and what it pulls in.

   The >=0.1.1 lower bound makes an unsupported interpreter fail loudly: releases from 0.1.1 declare Requires-Python >=3.10,<3.13, but 0.1.0 predates that bound, so on Python 3.13 a bare pip install psdn-sonar silently falls back to the outdated 0.1.0 — two releases behind, with no warning — instead of reporting that 3.13 is unsupported. With the bound, pip names the Requires-Python restriction and installs nothing.

3. Verify the install:

   psdn-sonar --version
   python -c "import psdn_sonar; print(psdn_sonar.__version__)"
   

Then follow docs/USAGE.md for runnable examples.

Contributor install (from source)

Contributors install the frozen, locked environment (exactly what CI runs).

macOS or Linux (bash/zsh):

git clone https://github.com/PSDN-AI/SONAR-OSS.git
cd SONAR-OSS
make setup            # uv sync --frozen with dev extras — does NOT include [ml]
# or, to run the docs/USAGE.md examples (local models, semantic similarity):
make setup-ml         # dev + [ml] extras, ~1.5 GB on disk
source .venv/bin/activate

Windows (make is not present on a stock host; the two targets are single uv commands, so run them directly — uv provisions the pinned CPython itself):

git clone https://github.com/PSDN-AI/SONAR-OSS.git
cd SONAR-OSS
uv sync --frozen --extra dev              # what `make setup` runs
# or, for the docs/USAGE.md examples:
uv sync --frozen --extra dev --extra ml   # what `make setup-ml` runs
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope Process
.\.venv\Scripts\Activate.ps1

(The execution-policy line is the same silent-refusal guard explained in the PyPI section; from cmd.exe use .venv\Scripts\activate.bat. The make targets themselves run on Windows from a Unix-style shell such as Git Bash, bundled with Git for Windows.)

Or with plain pip (editable, freshly resolved). Create and activate a virtual environment first — exactly as in step 1 of the PyPI install section above. On "externally managed" interpreters (PEP 668: Homebrew and Debian/Ubuntu system Pythons) pip otherwise refuses with error: externally-managed-environment and installs nothing:

pip install -e ".[dev]"       # contributor tooling only — no [ml]
pip install -e ".[dev,ml]"    # what the docs/USAGE.md examples need

The [ml] extra is what runs local HuggingFace models and POSEIDON's semantic similarity; without it the USAGE examples fail with a TypeError naming the extra. Add it to an existing make setup environment with uv pip install -e ".[ml]".

One model in the Bengali defaults needs more than [ml]: khushids_bengali is a PEFT/LoRA adapter and requires peft from the [bengali] extra (pip install "psdn-sonar[bengali]"). Without it, a --language bn run skips that model with a message naming the extra and evaluates the rest of the defaults; one unavailable model never aborts a multi-model run.

Note that pip does not read uv.lock: pip installs resolve dependency versions fresh from PyPI within the ranges in pyproject.toml, so they are not byte-for-byte reproducible the way uv sync --frozen is. This is the normal contract for downstream package installs; use uv when you need the locked contributor environment.

API keys and gated models

Copy .env.example to .env and fill in only the values you need (API keys are required only for optional hosted-model backends). For pyannote VAD/diarization, setting HF_TOKEN is not enough on its own: the pyannote models are gated on HuggingFace, so the token's account must also accept the user conditions on each model page — pyannote/segmentation-3.0, pyannote/speaker-diarization-3.1, and pyannote/speaker-diarization-community-1 (the diarization pipeline downloads the third as a gated dependency under pyannote.audio 4.x, even though no command names it) — otherwise runs fail with 403 ... not in the authorized list even though the token is valid.

The LLM-judged metrics (entity preservation and intent pass rate, importable from psdn_sonar.utils.llm_metrics — a library-only surface with no CLI subcommand) read GEMINI_API_KEY (preferred) or GOOGLE_API_KEY as an alternative. A .env entry and an exported variable both work, same as the other API keys; when the same name is set in both places, the exported variable wins, so a per-run env GEMINI_API_KEY=... psdn-sonar ... prefix overrides the checkout's .env.

Usage

See docs/USAGE.md for a short quickstart with runnable examples (scoring a pair, evaluating a model over a dataset, listing models), and docs/FAQ.md for common workflows: CLI commands, required input files, output layout, and how success is measured.

Before comparing published numbers, read docs/SCORE_INTERPRETATION.md: what the scores measure — multi-speaker results include preprocessing error, scores are comparable within a dataset only, and cells where a model is evaluated on its declared training corpus carry an in-domain marker.

Development

make lint              # ruff lint + format check
make typecheck         # ty type checker
make test              # pytest
make pre-commit-install
make check-internal-refs

On Windows, run these from a Unix-style shell such as Git Bash (bundled with Git for Windows): the recipes are POSIX shell, and GNU Make falls back to cmd when no Unix shell is on PATH.

See CONTRIBUTING.md for the full contributor guide, including PR title conventions and the import gate.

Community and links

• Website: psdn.ai
• X / Twitter: @psdnai
• LinkedIn: Poseidon
• Issues and feature requests: GitHub issues

License

MIT — see LICENSE.