SPAM

Code for Phone Segmentation and Recognition through Phonological Activation Mapping (SLT 2026).

Pretrained model: juice500/wavlm-24-phonemodel (WavLM-large, layer 24, fit on the TIMIT training set)

Installation

Requires Python 3.10+.

For inference:

pip install phonespam

To train or evaluate, clone the repo and install the train extra:

git clone https://github.com/juice500ml/phonespam.git
cd phonespam
uv venv && uv pip install -e ".[train]" && uv sync --group train
# or: python -m venv .venv && source .venv/bin/activate && pip install -e ".[train]"

Dataset preparation and evaluation also need phone-metrics. It is not on PyPI, and PyPI rejects direct URL requirements in published metadata, so it cannot be named in the train extra. uv sync --group train installs it; with plain pip:

pip install "phone-metrics @ git+https://github.com/stephenmac7/phone-metrics@v0.1.0"

Usage

from phonespam import PhoneModel, vocab_for_language

model = PhoneModel.from_pretrained("juice500/wavlm-24-phonemodel")  # device="cuda" for GPU

for seg in model.transcribe("utt.wav", vocab=vocab_for_language("eng")):
    print(f"{seg.start:.2f}-{seg.end:.2f}  {seg.label}")

transcribe runs the whole pipeline and returns contiguous Segment(start, end, label) tuples with times in seconds. vocab is optional: without it any PanPhon phone can be predicted; with it, output is restricted to that inventory.

The activation map

model.spam(...) returns the SPAM itself — the intermediate both the segmenter and the recognizer consume, one row per encoder frame and one column per phonological feature. It carries its own feature names and frame times, so pulling channels out for a plot doesn't mean tracking column positions:

spam = model.spam("utt.wav")  # normalized to [0, 1]
spam.shape  # (n_frames, n_features)
spam.featnames  # ("silence+", "syl+", "son+", ...)
spam.times  # frame start times in seconds

heat = spam.select(["silence+", "son+", "cons+", "strid+"])
plt.imshow(np.asarray(heat).T, aspect="auto", vmin=0, vmax=1)

spam.to_frame()  # pandas DataFrame indexed by time

np.asarray(spam) gives the bare matrix, and a Spam can be passed straight to model.recognize. Pass act="none" for the raw, unbounded projections instead of the sigmoid-normalized ones.

Segmentation only

model.boundary_times("utt.wav")  # array of boundary times in seconds

model.segment(features, waveform) is the lower-level call: it takes features you already have and returns frame indices rather than seconds.

Constraining the phone inventory

vocab_for_language resolves a language name, ISO 639-3 code, or Glottocode to that language's PHOIBLE inventory:

from phonespam import vocab_for_inventory, vocab_for_language

vocab = vocab_for_language("eng")  # or "deu", "stan1293", "German", ...
segs = model.transcribe("utt.wav", vocab=vocab)

# Some languages map to several PHOIBLE inventories; a warning names them all
# and one is chosen. Pick explicitly to silence it:
vocab = vocab_for_inventory(2175)  # Western/Mid-Western US English

# By default these are the inventory's *allophones*. Not every PHOIBLE source
# transcribes them; for those, ask for the phoneme inventory instead:
vocab = vocab_for_inventory(2252, phoneme=True)  # English (RP)

The inventory table is downloaded on first use; see License for where it is cached and how to point at your own copy.

Tuning the segmenter

hparam_overrides merges onto the model's stored hyperparameters for a single call. Segmenter.default_hparams() lists them all:

segs = model.transcribe("utt.wav", hparam_overrides={"combined_prominence": 0.005})
segs = model.transcribe("utt.wav", snap_silence=False)

For a sweep, copy the segmenter instead — the copy shares the posteriogram weights, so nothing is refit:

wav = model.load_audio("utt.wav")
feats = model.extract_features(wav)

tuned = model.segmenter.with_hparams({"distance": "l2"})
boundaries = tuned.segment(feats, wav)  # frame indices

Driving the stages yourself

Two conventions are easy to miss here, and both fail silently — the output keeps the right shape and only the values are wrong. transcribe exists to encapsulate them, and recognize warns if it detects either:

• Segmenter.segment returns frame indices; Recognizer.recognize takes times in seconds. PhoneModel.segment/recognize handle the conversion.
• The recognizer scores a sigmoid posteriogram, but PhonologicalPosteriogram.project defaults to act="none".

wav = model.load_audio("utt.wav")  # mono float32, resampled to the model's rate
feats = model.extract_features(wav)  # (T, D) S3M features — one encoder pass
spam = model.spam_from_features(feats)  # reuses feats, no second pass

boundaries = model.segment(feats, wav)  # frame indices
times = model.encoder.frame_to_time(boundaries)  # seconds
phones = model.recognize(spam, boundaries, vocab=vocab_for_language("eng"))

model.sr, model.frame_shift  # 16000, 320 — one frame is frame_shift/sr seconds

scripts/demo_LDC93S1.py is a worked example: it runs one utterance four ways (reference, per-frame, reference-boundaries, full pipeline) and plots them against the activation map.

Saving and sharing a model

model.to("cuda")
model.save_pretrained("exp/my-phonemodel")  # writes model.pt
model.push_to_hub("me/my-phonemodel", private=True)  # needs `huggingface-cli login`

Public API at a glance

Training and evaluation

Data

• TIMIT is distributed by the LDC (LDC93S1). Place the root of the official distribution at data/TIMIT.

• VoxAngeles (evaluation only):

  git clone --branch main --depth 1 https://github.com/pacscilab/voxangeles.git data/voxangeles
  (cd data/voxangeles/data/audited_aligned && for f in *.zip; do unzip -o "$f"; done)
  

Train the released model

# 1. Per-phone CSVs. Training uses timit-raw.csv, which keeps stop closures.
python -m phonespam.training.prepare_datasets \
    --dataset_type timit --dataset_path data/TIMIT --output_dir data/csv

# 2. Per-phone S3M features for the training split.
python -m phonespam.training.extract_features \
    --model microsoft/wavlm-large --layer_index 24 --pool center --sr 16000 \
    --dataset_csv data/csv/timit-raw.csv --split train \
    --device cuda:0 --output_path exp/wavlm-24.feats.pkl

# 3. Fit the phonological vectors and save exp/wavlm-24-phonemodel/model.pt.
python -m phonespam.training.train \
    --features_pkl exp/wavlm-24.feats.pkl --output_dir exp/wavlm-24-phonemodel

Evaluate

python scripts/evaluate_metrics.py --model exp/wavlm-24-phonemodel \
    --timit_root data/TIMIT --vox_root data/voxangeles

--model also accepts a Hub id such as juice500/wavlm-24-phonemodel. For TIMIT test and VoxAngeles, the script reports boundary R-value (20 ms, strict) and PER/TER/PFER with both ground-truth ("oracle") and predicted ("pipeline") segmentation. On VoxAngeles, oracle PFER is also split into phones seen and unseen in TIMIT. Set CACHE_DIR=.cache/eval to cache per-utterance inference across runs.

Development

uv pip install -e ".[train,dev]"
pre-commit install
pytest

Citation

@inproceedings{spam2026,
  title     = {Phone Segmentation and Recognition through Phonological Activation Mapping},
  author    = {Bharadwaj, Shikhar and Choi, Kwanghee and McIntosh, Stephen and Li, Chin-Jou and
               Yeo, Eunjung and Saito, Daisuke and Minematsu, Nobuaki and Watanabe, Shinji and
               Zhu, Jian and Harwath, David and Mortensen, David R.},
  booktitle = {Proc. IEEE Spoken Language Technology Workshop (SLT)},
  year      = {2026}
}

License

The code is MIT-licensed. The phone inventory table it downloads at runtime is PHOIBLE 2.0 data, licensed separately under CC BY-SA 3.0:

Moran, Steven & McCloy, Daniel (eds.) 2019. PHOIBLE 2.0. Jena: Max Planck Institute for the Science of Human History. DOI: 10.5281/zenodo.2626687

The table is a ~26 MB CSV and is not bundled in the wheel. It is downloaded once from a pinned, checksum-verified upstream commit and cached under ~/.cache/phonespam/ (or $XDG_CACHE_HOME/phonespam). Set PHONESPAM_CACHE_DIR to relocate the cache, or PHONESPAM_PHOIBLE_CSV to use a copy you already have (and skip the download entirely).