pitchbench 0.1.6

Benchmark suite for evaluating pitch and acoustic perception in audio language models

PitchBench -- Python Package

Benchmark suite for evaluating pitch and acoustic perception in audio language models (ALMs). Probes pitch identification, temporal localisation, chord recognition, melodic contour, robustness to audio effects, and more — reporting per-format accuracy (MIDI, SPN, doremi, Hz) to expose where verbal decoding fails.

---

Setup

1. Install FluidSynth

Platform	Command
Linux / WSL	sudo apt install fluidsynth
macOS	brew install fluid-synth
Windows	choco install fluidsynth or download from fluidsynth.org

A GM soundfont is also required. Place any .sf2 or .sf3 in data/soundfonts/ — no further configuration needed.

Linux / WSL:

sudo apt install fluid-soundfont-gm
mkdir -p data/soundfonts
cp /usr/share/sounds/sf2/FluidR3_GM.sf2 data/soundfonts/

macOS / cross-platform:

curl -L "http://ftp.debian.org/debian/pool/main/f/fluid-soundfont/fluid-soundfont-gm_3.1-5.3_all.deb" \
     -o /tmp/fluid.deb
cd /tmp && ar x fluid.deb && tar xf data.tar.* ./usr/share/sounds/sf2/FluidR3_GM.sf2
mkdir -p data/soundfonts && cp /tmp/usr/share/sounds/sf2/FluidR3_GM.sf2 data/soundfonts/

SF2 resolution order: PITCHBENCH_SF2 env var → data/soundfonts/ → /usr/share/sounds/sf2/FluidR3_GM.sf2.

2. Install PitchBench

python3 -m venv .venv
source .venv/bin/activate        # Linux / macOS
# .venv\Scripts\activate         # Windows
pip install pitchbench

For development tools:

uv sync --extra dev              # installs pytest + pytest-mock

Requires Python ≥ 3.12.

3. Configure a model backend

OpenRouter — add to .env:

OPENROUTER_KEY=sk-or-...

Use --model openrouter/<provider>/<model>. "Latest" alias slugs require a ~ prefix, e.g. openrouter/~google/gemini-flash-latest.

DashScope — add to .env:

DASHSCOPE_API_KEY=sk-...

Use --model dashscope/<model>.

Local server — included in the base pip install pitchbench dependencies:

--model http://localhost:8001 --name my-local-model

---

Usage

PitchBench separates stimulus generation from model evaluation. Generate once, evaluate any number of models.

Generate

pitchbench generate all
pitchbench generate a          # single category
pitchbench generate a1         # single experiment

Writes WAV files and a Parquet dataset to data/generated/<exp>/. Re-running is safe: existing rows are skipped. The default set of experiment parameters generates more than 100,000 audio files, which can take several hours. The dataset can be easily extended by allowing greater coverage by the experiment parameters through a custom experiment config. Run:

pitchbench generate all --source your_config.py

An example config can be found on the original Github repository of this package called PitchBench.

data/generated/pitchbench_a1_single_pitch_id/
  *.wav               # one file per condition
  _questions.parquet  # HuggingFace-compatible schema
  _questions.csv      # human-readable companion

Evaluate

pitchbench --list   
pitchbench evaluate all --model openrouter/<provider>/<model>
pitchbench evaluate a1  --model openrouter/<provider>/<model>

# Quick test (20 stimuli, stratified)
pitchbench evaluate a1  --model openrouter/<provider>/<model> --sample-n 20 --sample-seed 0

Results land in results/evaluation/<model_slug>/<YYYYMMDD_HHMMSS>/.

In tmux (recommended for long runs):

tmux new-session -d -s mymodel "cd /home/hsnu2/PitchBench && source .venv/bin/activate && \
  pitchbench evaluate all --model openrouter/<provider>/<model> 2>&1 | tee logs/eval_mymodel_\$(date +%Y%m%d_%H%M%S).log"

Analyze

pitchbench analyze --preset q1 --model openrouter/<provider>/<model>

Writes to results/analysis/<model_slug>/<run>/ and generates a summary of the ablations.

Batch (multiple models):

python -m pitchbench.analysis.run_analysis q1 \
    --models openrouter/<provider>/<model1> \
             openrouter/<provider>/<model2> \
             dashscope/<model3>

---

Output

results/evaluation/<model_slug>/<YYYYMMDD_HHMMSS>/
  pitchbench_<exp>/
    results_<model>.json      # metadata, git commit, per-item responses
    results_<model>.csv       # one row per stimulus
    accuracies_<model>.csv    # aggregate metrics
    plots/                    # per-format CI plots

results/evaluation/summary/ is populated automatically after each run (aggregated_accuracies.csv, accuracy plots by instrument / notation / note).

---

Experiment categories

28 experiments across 7 categories.

Category A — Single-pitch identification

ID	Script	What it tests
a1	single_pitch_id	Baseline: identify one sustained note across all sources and formats
a2	single_pitch_by_loudness	Pitch accuracy under different loudness levels
a3	single_pitch_by_duration	Pitch accuracy under different durations

Category B — Time-localised pitch

ID	Script	What it tests
b1	single_pitch_within_silence	Hidden note in a silent clip
b2	pitch_at_timestamp	Which pitch is playing at a queried timestamp in a multi-note sequence
b3	timestamp_single_pitch	Detect when a single note starts/ends
b4	timestamp_specific_pitch	Onset/offset of a named target pitch among distractors
b5	timestamp_multiple_pitches	Full timing transcription of all notes in a sequence

Category C — Chords and simultaneous pitches

ID	Script	What it tests
c1	chord_count_pitches	Count distinct pitches in a chord (dyads, triads, 7ths)
c2	chord_dyad_interval	Name the interval (semitone count) between two simultaneous tones
c3	chord_quality	Name chord quality and/or root+quality from a sounding chord
c4	chord_pitches	List every note in a chord

Category D — Sequences, contour, intervals

ID	Script	What it tests
d1	sequence_count_pitches	Count distinct pitches in a sequential passage
d2	dyad_lower_higher_difference	Binary higher/lower judgment
d3	contour_discrete	Output up/down tokens for each step-wise transition
d4	contour_continuous	Output up/down tokens for each monotonic movement
d5	sequence_ranking_by_pitch	Rank sequential tones from lowest to highest
d6	sequence_dyad_interval	Name the melodic interval between two sequential notes
d7a	pitch_with_reference	Pitch identification given a labelled reference tone (variant used in the evaluation)
d7b	pitch_with_reference_split	Same, but reference and target in separate audio clips
d7c	pitch_with_reference	Variant with different set of pitches
d7d	pitch_with_reference_split	Split-clip variant with a different set of pitches
d8	sequence_pitches	Transcribe all pitches in a note sequence in order

Category E — Robustness

ID	Script	What it tests
e1	audio_effects	Pitch under effects like high-pass/low-pass filtering, distortion, reverb, chorus
e2	background	Pitch over real-world backgrounds (crowd, rain, bells, street) at varying loudness levels
e3	harmonic_saturation	Pitch with harmonic saturation / overdrive
e4	time_stretching	Pitch with time-stretching (speed change without pitch shift)
e5	vibrato	Pitch with vibrato at varying rates and depths
e6	slightly_off	Detuned tones (up to 45 % of a semitone)

Category F — Polyphony

ID	Script	What it tests
f1	melodic_line_atonal	Transcribe one designated voice from 2–3 simultaneous synthetic voices (atonal)
f2	melodic_line_tonal	Same task over tonal melodic material from Bach chorales

Category Y — Format variants

ID	Script	What it tests
y1	single_pitch_id_mcq	MCQ variant of a1: pick the correct pitch from labelled options

---

Stimulus engine

• Waveforms (always available): sine, sawtooth, square, triangle
• GM instruments (FluidSynth): piano, electric_keyboard, guitar, flute, trumpet, trombone, clarinet, oboe, violin, cello, organ, bass, synth_lead, synth_pad, voice
• Backgrounds (e2): white_noise synthesised; real recordings in data/preloaded/background/<name>.mp3

---

Repository layout

src/pitchbench/
  config.py                    # runtime constants and env wiring
  configs/
    benchmark_config.py        # paper-locked data-gen parameters
    analysis_config.py         # analysis-mode presets/overrides
    plot_config.py             # labels/colors for figures
    user_config.py             # user-overridable settings
  sound/
    engine.py                  # central audio synthesis engine
  model/
    query.py                   # ALM query facade (local/OpenRouter/DashScope)
    dispatcher.py              # provider-aware request dispatch
    cost.py                    # API usage/cost tracking
  experiments/
    run.py                     # pitchbench CLI
    scripts/                   # one .py per experiment (32 total)
    helpers/
      cat_a.py ... cat_f.py    # per-category generate + evaluate helpers
      audit.py                 # condition/result audit utilities
      data.py                  # Parquet I/O
      music.py                 # pitch/notation conversion helpers
      plots.py                 # experiment plotting helpers
      results.py               # result writers, aggregators, summary CSVs
      sampling.py              # stratified sampling
      setup.py                 # experiment setup helpers
      timing_layout.py         # timing-grid utilities
  analysis/
    analyze_a1.py              # a1 line plots + heatmaps
    analyze.py                 # core analysis pipeline
    a1.py                      # alternate A1 analysis entrypoint
    ablation.py                # ablation summaries
    combine.py                 # combine multi-run CSV outputs
    overview.py                # overview plots/tables
    run_analysis.py            # batch analysis CLI
data/
  preloaded/                   # background recordings (gitignored)
  generated/                   # created by `pitchbench generate`
results/                       # created by evaluate/analyze runs

Set PITCHBENCH_ROOT to override the project root for data/ and results/.

---

Environment variables

Variable	Default	Purpose
PITCHBENCH_ROOT	.	Project root (data/, results/)
PITCHBENCH_SF2	auto-discovered	GM soundfont path override
PITCHBENCH_LOCAL_URL	http://localhost:8001	Default local model server
OPENROUTER_KEY	—	OpenRouter API key
PITCHBENCH_CONCURRENCY_OPENROUTER	20	Max parallel OpenRouter calls
PITCHBENCH_CONCURRENCY_DASHSCOPE	10	Max parallel DashScope calls
LOCAL_CONCURRENCY	2	Max parallel local server calls

---

Tests

uv run pytest

77 tests covering CLI resolution, Parquet I/O, result aggregation, sampling, and API routing. No network calls or audio generation required.

License

MIT — see LICENSE.