Skip to main content

A lightweight library to compute Diarization Error Rate (DER).

Project description

SimpleDER Build Status PyPI Version Python Versions Documentation

Overview

This is a lightweight library to compute Diarization Error Rate (DER).

Features NOT supported:

  • Handling overlapped speech, i.e. two speakers speaking at the same time.
  • Allowing segment boundary tolerance, a.k.a. the collar value.

For more sophisticated metrics with these supports, please use pyannote-metrics instead.

To learn more about speaker diarization, here is a curated list of resources: awesome-diarization.

Diarization Error Rate

Diarization Error Rate (DER) is the most commonly used metrics for speaker diarization.

Its strict form is:

       False Alarm + Miss + Overlap + Confusion
DER = ------------------------------------------
                   Reference Length

The definition of each term:

  • Reference Length: The total length of the reference (ground truth).
  • False Alarm: Length of segments which are considered as speech in hypothesis, but not in reference.
  • Miss: Length of segments which are considered as speech in reference, but not in hypothesis.
  • Overlap: Length of segments which are considered as overlapped speech in hypothesis, but not in reference. This library does NOT support overlap.
  • Confusion: Length of segments which are assigned to different speakers in hypothesis and reference (after applying an optimal assignment).

The unit of each term is seconds.

Note that DER can theoretically be larger than 1.0.

References:

Tutorial

Install

Install the package by:

pip3 install simpleder

or

python3 -m pip install simpleder

API

Here is a minimal example:

import simpleder

# reference (ground truth)
ref = [("A", 0.0, 1.0),
       ("B", 1.0, 1.5),
       ("A", 1.6, 2.1)]

# hypothesis (diarization result from your algorithm)
hyp = [("1", 0.0, 0.8),
       ("2", 0.8, 1.4),
       ("3", 1.5, 1.8),
       ("1", 1.8, 2.0)]

error = simpleder.DER(ref, hyp)

print("DER={:.3f}".format(error))

This should output:

DER=0.350

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
simpleder-0.0.2-py3-none-any.whl (8.3 kB) Copy SHA256 hash SHA256 Wheel py3
simpleder-0.0.2.tar.gz (3.9 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page