Skip to main content

Evaluate your speech-to-text system with similarity measures such as word error rate (WER)

Project description

JiWER

JiWER is a simple and fast python package to evaluate an automatic speech recognition system. It supports the following measures:

  1. word error rate (WER)
  2. match error rate (MER)
  3. word information lost (WIL)
  4. word information preserved (WIP)
  5. character error rate (CER)

These measures are computed with the use of the minimum-edit distance between one or more reference and hypothesis sentences. The minimum-edit distance is calculated using RapidFuzz, which uses C++ under the hood, and is therefore faster than a pure python implementation.

Documentation

For further info, see the documentation at jitsi.github.io/jiwer.

Installation

You should be able to install this package using uv:

$ uv add jiwer

Or, if you prefer old-fashioned pip and you're using Python >= 3.8:

$ pip install jiwer

Usage

The most simple use-case is computing the word error rate between two strings:

from jiwer import wer

reference = "hello world"
hypothesis = "hello duck"

error = wer(reference, hypothesis)

A note on empty references

There is undefined behaviour when you apply an empty reference and hypothesis pair to the WER formula, as you get a division by zero.

As of version 4.0, jiwer defines the behaviour as follows, and thereby also lifts the requirement for reference strings to be non-empty. This allows for testing whether models hallucinate on silent audio. Note that usually, there are multiple reference and hypothesis pairs. It now supported that one or more of these references are empty, but to test well, most references should still be non-empty.

import jiwer

# when ref and hyp are both empty, there is no error as
# an ASR system correctly predicted silence/non-speech.
assert jiwer.wer('', '') == 0 
assert jiwer.mer('', '') == 0
assert jiwer.wip('', '') == 1
assert jiwer.wil('', '') == 0

assert jiwer.cer('', '') == 0

When the hypothesis is non-empty, every word or character counts as an insertion:

import jiwer

assert jiwer.wer('', 'silence') == 1
assert jiwer.wer('', 'peaceful silence') == 2
assert jiwer.process_words('', 'now defined behaviour').insertions == 3

assert jiwer.cer('', 'a') == 1
assert jiwer.cer('', 'abcde') == 5

Licence

The jiwer package is released under the Apache License, Version 2.0 licence by 8x8.

For further information, see LICENCE.

Reference

For a comparison between WER, MER and WIL, see:
Morris, Andrew & Maier, Viktoria & Green, Phil. (2004). From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

jiwer-4.0.0.tar.gz (28.1 kB view details)

Uploaded Source

Built Distribution

jiwer-4.0.0-py3-none-any.whl (23.0 kB view details)

Uploaded Python 3

File details

Details for the file jiwer-4.0.0.tar.gz.

File metadata

  • Download URL: jiwer-4.0.0.tar.gz
  • Upload date:
  • Size: 28.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: uv/0.5.26

File hashes

Hashes for jiwer-4.0.0.tar.gz
Algorithm Hash digest
SHA256 ae9c051469102a61ef0927100baeeb4546f78d180c9b0948281d08eaf44c191e
MD5 aae45c578e176de9509bda73069cf5af
BLAKE2b-256 121e963dfc249d5bbe88079b57a22556e981ddd9208e4b6116e48bdbfc01f26b

See more details on using hashes here.

File details

Details for the file jiwer-4.0.0-py3-none-any.whl.

File metadata

  • Download URL: jiwer-4.0.0-py3-none-any.whl
  • Upload date:
  • Size: 23.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: uv/0.5.26

File hashes

Hashes for jiwer-4.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 7efaf0bd336b095d99ddef9dd67e1ee829d75d58aa2a81d9639870b01d6d95ea
MD5 60774d806682b6c51995f3ae4af94853
BLAKE2b-256 69c9172c525330c739a068c01050759a6f855ce16212db10a0359e690a03ac48

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page