SimpleDER

Table of Contents

• Overview
• Diarization Error Rate
• Jaccard Error Rate
• Implementation
• Tutorial
• Citation

Overview

This is a single-file lightweight library to compute speaker diarization evaluation metrics:

• Diarization Error Rate (DER) with full component breakdown (Miss, False Alarm, Confusion) and optimal speaker mapping
• Jaccard Error Rate (JER) with per-speaker breakdown and optimal speaker mapping
• Overlapped speech support, plus optional collar tolerance and skip_overlap filtering
• Corpus-level evaluation (corpus_der and corpus_jer) across multiple recordings

To learn more about speaker diarization, here is a curated list of resources: awesome-diarization.

Diarization Error Rate

Diarization Error Rate (DER) is the most commonly used metric for speaker diarization:

       False Alarm + Miss + Confusion
DER = --------------------------------
              Reference Length

The definition of each term (all measured in seconds, accounting for overlapped speech):

• Reference Length: Total speaker-time duration in the reference (ground truth), $\int N_{\text{ref}}(t) , dt$.
• False Alarm: Duration of hypothesis speakers in excess of reference speakers, $\int \max(0, N_{\text{hyp}}(t) - N_{\text{ref}}(t)) , dt$.
• Miss: Duration of reference speakers missed by the hypothesis, $\int \max(0, N_{\text{ref}}(t) - N_{\text{hyp}}(t)) , dt$.
• Confusion: Duration of active speakers assigned to the wrong speaker after applying an optimal 1-to-1 speaker mapping.

Note that DER can be larger than 1.0.

References:

• pyannote-metrics documentation
• Xavier Anguera's thesis

Jaccard Error Rate

Jaccard Error Rate (JER), introduced for the DIHARD II challenge, evaluates diarization with equal weight for each reference speaker. Under an optimal 1-to-1 mapping between reference speakers $r \in S_{\text{ref}}$ and hypothesis speakers $h \in S_{\text{hyp}}$:

$$\text{JER} = \frac{1}{|S_{\text{ref}}|} \sum_{r \in S_{\text{ref}}} \left(1 - \frac{\text{duration}(r \cap h)}{\text{duration}(r \cup h)}\right)$$

Unmapped reference speakers receive a speaker error of 1.0. Unlike DER, JER is always bounded in [0.0, 1.0].

Implementation

1. Per-Speaker Canonicalization: Overlapping or contiguous segments belonging to the same speaker are merged so each speaker's timeline consists of disjoint intervals.
2. Pre-processing (collar and skip_overlap):
   • If collar > 0: For every reference boundary $t$, the interval $[t - \text{collar}, t + \text{collar}]$ (NIST md-eval convention) is excluded from both reference and hypothesis.
   • If skip_overlap=True: Regions where $N_{\text{ref}}(t) > 1$ are excluded from both reference and hypothesis.
3. Optimal Mapping: We solve the linear sum assignment problem via the Hungarian algorithm (scipy.optimize.linear_sum_assignment)—maximizing total overlap duration $\text{Match}$ for DER, or total Jaccard similarity for JER.
4. Error & Load Calculation: Let $\text{Load} = \int \max(N_{\text{ref}}(t), N_{\text{hyp}}(t)) , dt$. Then: $$\text{Miss} + \text{False Alarm} + \text{Confusion} = \text{Load} - \text{Match}$$ $$\text{DER} = \frac{\text{Load} - \text{Match}}{\text{Reference Length}}$$

Tutorial

Install

pip3 install simpleder

API

Basic DER and JER

import simpleder

# reference (ground truth)
ref = [("A", 0.0, 1.0),
       ("B", 1.0, 1.5),
       ("A", 1.6, 2.1)]

# hypothesis (diarization result from your algorithm)
hyp = [("1", 0.0, 0.8),
       ("2", 0.8, 1.4),
       ("3", 1.5, 1.8),
       ("1", 1.8, 2.0)]

der = simpleder.DER(ref, hyp)
jer = simpleder.JER(ref, hyp)
print(f"DER={der:.3f}, JER={jer:.3f}")
# DER=0.350, JER=0.381

Options: collar, skip_overlap, and detailed Breakdown

details = simpleder.DER(ref, hyp, collar=0.0, skip_overlap=False, detailed=True)
# {
#   'der': 0.35,
#   'miss': 0.2,
#   'false_alarm': 0.1,
#   'confusion': 0.4,
#   'ref_length': 2.0,
#   'speaker_mapping': {'A': '1', 'B': '2'}
# }

Corpus-Level Evaluation (Multiple Recordings)

corpus_der computes the micro-averaged DER across recordings, and corpus_jer averages per-speaker JER across all reference speakers in the corpus (accepting either dict or list inputs):

refs = {"rec1": ref, "rec2": [("A", 0.0, 2.0)]}
hyps = {"rec1": hyp, "rec2": [("1", 0.0, 2.0)]}

total_der = simpleder.corpus_der(refs, hyps)
total_jer = simpleder.corpus_jer(refs, hyps)

Citation

We developed this package as part of the following work:

@inproceedings{wang2018speaker,
  title={{Speaker Diarization with LSTM}},
  author={Wang, Quan and Downey, Carlton and Wan, Li and Mansfield, Philip Andrew and Moreno, Ignacio Lopz},
  booktitle={2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={5239--5243},
  year={2018},
  organization={IEEE}
}

@inproceedings{xia2022turn,
  title={{Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detection}},
  author={Wei Xia and Han Lu and Quan Wang and Anshuman Tripathi and Yiling Huang and Ignacio Lopez Moreno and Hasim Sak},
  booktitle={2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={8077--8081},
  year={2022},
  organization={IEEE}
}

@inproceedings{huang24d_interspeech,
  title={{On the Success and Limitations of Auxiliary Network Based Word-Level End-to-End Neural Speaker Diarization}},
  author={Yiling Huang and Weiran Wang and Guanlong Zhao and Hank Liao and Wei Xia and Quan Wang},
  year={2024},
  booktitle={Interspeech 2024},
  pages={32--36},
}