AiDA Whisper Evaluation Framework (Serbian)
An evaluation framework for Serbian Whisper models.
Whisper Evaluator 🎤
A simple, modular framework to evaluate fine-tuned Whisper models in Python notebooks.
This library allows you to easily run evaluations on any dataset from the Hugging Face Hub using a simple configuration dictionary. It calculates a comprehensive set of metrics, including WER, CER, BLEU, and ROUGE, and automatically logs all results to a file.
Installation
You can install the library directly from GitHub for latest updates and features. Make sure you have git installed on your system.
pip install git+https://github.com/your-username/whisper-evaluator.git
Quickstart
Using the library in a Google Colab or Jupyter Notebook is straightforward.
from whisper_evaluator import Evaluator
import json
# 1. Define your evaluation configuration
config = {
"model_args": {
"name_or_path": "openai/whisper-large-v2", # Your fine-tuned model ID
"device": "cuda"
},
"task_args": {
"dataset_name": "mozilla-foundation/common_voice_11_0",
"dataset_subset": "sr", # Serbian language
"dataset_split": "test[:20]", # Use the first 20 samples for a quick demo
"audio_column": "audio",
"text_column": "sentence"
}
}
# 2. Initialize the evaluator
evaluator = Evaluator(config=config)
# 3. Run the evaluation (logs to 'evaluation_log.txt' by default)
detailed_results, metrics = evaluator.run()
# 4. Analyze the results
print("\n--- Final Metrics ---")
# Pretty print the metrics dictionary
print(json.dumps(metrics, indent=2))
print("\n--- Sample of evaluation details ---")
# Print the first 3 results from the list
for i, result in enumerate(detailed_results[:3]):
print(f"\n--- Example {i+1} ---")
print(f"Reference: {result['reference']}")
print(f"Prediction: {result['prediction']}")
Project Setup
Follow these steps to set up the AiDA-Whisper-Eval project.
Using Conda
1. Create a new Conda environment
conda create --name aida python=3.12 -y
conda activate aida
2. Install Poetry
pip install poetry
3. Install project dependencies
Navigate to the project's root directory and run:
poetry install
Using Plain Python
1. Create and activate a virtual environment
python -m venv venv
# On Linux/macOS
source venv/bin/activate
# On Windows
.\venv\Scripts\activate
2. Upgrade pip and install Poetry
pip install --upgrade pip
pip install poetry
3. Install project dependencies
From the project's root directory, run:
poetry install
pip install pre-commit
4. Set up pre-commit hooks
poetry run pre-commit install
Verifying Installation
Check installation by running tests:
# On Linux/macOS
make test
# On Windows
poetry run pytest
Your setup is complete!
1 maintainer
Metadata
1 maintainer
Release files for whisper-eval-serbian 0.0.35
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| whisper_eval_serbian-0.0.35.tar.gz | 15.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| whisper_eval_serbian-0.0.35-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 29.9 kB
Release files / whisper_eval_serbian-0.0.35.tar.gz
| Download URL | whisper_eval_serbian-0.0.35.tar.gz |
|---|---|
| Size | 15.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e0111ee7f2ce8b04abea1724231cde1a40f64286289d8a40fc6ef17ef177cdbd
|
|
BLAKE2b-256 checksum How to use checksums |
0c7e38e756d50df85b210a65d04d1cef469e3f8a7df970d5211270c41f671448
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.1.4 CPython/3.12.11 Linux/6.14.0-28-generic
|
Release files / whisper_eval_serbian-0.0.35-py3-none-any.whl
| Download URL | whisper_eval_serbian-0.0.35-py3-none-any.whl |
|---|---|
| Size | 14.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e8a708bead7da1b04ec1d1014b4fd88b7f05755daa940a9ae0e4f9e88e76c1df
|
|
BLAKE2b-256 checksum How to use checksums |
0636985952d3cbc5c732e97a9e5e11e40f9902709274408e5a8120aaa6a00727
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.1.4 CPython/3.12.11 Linux/6.14.0-28-generic
|
