DeafBench
Evaluate what ASR metrics miss.
DeafBench is an open-source benchmark for measuring AI caption failures that matter to Deaf and hard-of-hearing users.
Traditional ASR benchmark:
Model A: 7.8% WER
Model B: 8.1% WER
DeafBench:
Model A: 72% critical information preserved
Model B: 96% critical information preserved
Two models can have similar Word Error Rate (WER) while producing very different accessibility outcomes. DeafBench highlights critical information loss (names, numbers, negations, dates, technical terms) and non-speech annotations that traditional WER obscures.
Quickstart
Installation
pip install deafbench
Or install locally for development:
git clone https://github.com/488315/DeafBench.git
cd DeafBench
pip install -e .
Usage
1. Compare predictions against reference captions:
deafbench compare examples/references.jsonl examples/model-a.jsonl
Output:
DeafBench v0.1
Samples: 3
WER 8.4%
Critical Information 91.2%
Non-Speech Information 62.5%
Speaker Attribution 87.0%
Median Latency 1.4s
⚠ 2 critical-information failures detected
2. Generate a Markdown evaluation report:
deafbench report examples/references.jsonl examples/model-a.jsonl --output report.md
Input JSONL Schema
references.jsonl
{
"id": "sample-001",
"text": "John Doe needs 25 milligrams on Friday.",
"critical": ["John Doe", "25 milligrams", "Friday"],
"sounds": ["[alarm]"],
"speaker": "Speaker 1"
}
predictions.jsonl
{
"id": "sample-001",
"text": "Guy needs 20 milligrams on Friday.",
"latency_ms": 820,
"speaker": "Speaker 1"
}
License
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file deafbench-0.1.1.tar.gz.
File metadata
- Download URL: deafbench-0.1.1.tar.gz
- Upload date:
- Size: 14.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
3207b00f92d4fb8b88c29894ea8fdf340fc3498d854b7c903e99d62816647eac
|
|
| MD5 |
28759e795f80f1b11a63ddf9f8f52db4
|
|
| BLAKE2b-256 |
f17cafe185c7ae00c6174bdd777a94ac4940443bd42a73a1ab6019089ab8433a
|
Provenance
The following attestation bundles were made for deafbench-0.1.1.tar.gz:
Publisher:
release.yml on 488315/DeafBench
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
deafbench-0.1.1.tar.gz -
Subject digest:
3207b00f92d4fb8b88c29894ea8fdf340fc3498d854b7c903e99d62816647eac - Sigstore transparency entry: 2368592891
- Sigstore integration time:
-
Permalink:
488315/DeafBench@3aa0d54165d880f920ac3308065dfe04c3d2edb9 -
Branch / Tag:
refs/heads/main - Owner: https://github.com/488315
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@3aa0d54165d880f920ac3308065dfe04c3d2edb9 -
Trigger Event:
workflow_dispatch
-
Statement type:
File details
Details for the file deafbench-0.1.1-py3-none-any.whl.
File metadata
- Download URL: deafbench-0.1.1-py3-none-any.whl
- Upload date:
- Size: 17.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
201726347cdaf4b0784cbfc282cbda26c10c651af97e3c7ac5ebf076f06e4469
|
|
| MD5 |
5b49cd770ce167163fdd8de662a52093
|
|
| BLAKE2b-256 |
9b3821d08f0fe4549d499619a7003efea0c5dad18dff67eecd7ad807df8c8ff2
|
Provenance
The following attestation bundles were made for deafbench-0.1.1-py3-none-any.whl:
Publisher:
release.yml on 488315/DeafBench
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
deafbench-0.1.1-py3-none-any.whl -
Subject digest:
201726347cdaf4b0784cbfc282cbda26c10c651af97e3c7ac5ebf076f06e4469 - Sigstore transparency entry: 2368593077
- Sigstore integration time:
-
Permalink:
488315/DeafBench@3aa0d54165d880f920ac3308065dfe04c3d2edb9 -
Branch / Tag:
refs/heads/main - Owner: https://github.com/488315
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@3aa0d54165d880f920ac3308065dfe04c3d2edb9 -
Trigger Event:
workflow_dispatch
-
Statement type:
