A python package for whisper normalizer
Project description
whisper_normalizer
Installation of package
pip install whisper_normalizer
or from github repository
pip install git+https://github.com/kurianbenoy/whisper_normalizer.git
Why should we normalize/standardize text?
- In ASR systems it’s important to normalize the text to reduce error in metrics like WER, CER etc.
- Text normalization/standardization is process of converting texts in different styles into a standardized form, which is a best-effort attempt to penalize only when a word error is caused by actually mistranscribing a word, and not by formatting or punctuation differences.(from Whisper paper)
Why use this python package?
This package is a python implementation of the text standardisation/normalization approach which is being used in OpenAI whisper text normalizer. If you want to use just text normalization alone, it’s better to use this instead reimplementing the same thing. OpenAI approach of text normalization is very helpful and is being used as normalization step when evaluating competitive models like AssemblyAI Conformer-1 model.
Models evaluated using Whisper normalization
- Massively Multilingual Speech (MMS) models by Meta
- Conformer 1 by AssemblyAI
- Conformer 2 by AssemblyAI
How to use
OpenAI open source approach of text normalization/standardization is mentioned in detail Appendix Section C pp.21 the paper Robust Speech Recognition via Large-Scale Weak Supervision.
You can use the same thing in this package as follows:
from whisper_normalizer.basic import BasicTextNormalizer
normalizer = BasicTextNormalizer()
normalizer("I'm a little teapot, short and stout. Tip me over and pour me out!")
'i m a little teapot short and stout tip me over and pour me out '
from whisper_normalizer.english import EnglishTextNormalizer
english_normalizer = EnglishTextNormalizer()
english_normalizer("I'm a little teapot, short and stout. Tip me over and pour me out!")
'i am a little teapot short and stout tip me over and pour me out'
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for whisper_normalizer-0.0.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 33848c78257852b6101998f45e4bbfed14b1acbafb8836f57d9a67c05f0314dd |
|
MD5 | f32f264775d572543bb834b63a36b8b2 |
|
BLAKE2b-256 | 104cff64e293e0bfefb2af86fa5728e89efb17c538ed906b6894ae7fff92056b |