script-fidelity

script-fidelity is a small Python package for Script Fidelity Rate (SFR), a reference-free metric for multilingual ASR. SFR measures the fraction of countable hypothesis characters that belong to the expected Unicode script for a target language.

Quick signals:

• Install with uv add script-fidelity
• Load with HF Evaluate via themechanism/script_fidelity_rate
• Supports 102 FLEURS language configs, excluding all
• PyPI: https://pypi.org/project/script-fidelity/

Use SFR with WER and CER. SFR checks script validity; WER and CER measure transcription error against references.

install

For package development in this repo:

uv sync --extra dev

For a downstream project:

uv add script-fidelity

Run the CLI without adding it to a project:

uvx --from script-fidelity sfr score --language ps_af --text "کابل کې ښه هوا ده"

python use

from script_fidelity import compute_sfr, compute_sfr_batch

score = compute_sfr("کابل کې ښه هوا ده", language="ps_af")
scores = compute_sfr_batch(
    ["کابل کې ښه هوا ده", "this is romanized output"],
    language="pashto",
)

Digits count by default, matching the paper. Treat digits as neutral with digit_policy="ignore".

compute_sfr("کابل 2026", language="ps_af", digit_policy="ignore")

HF Evaluate use

Local metric:

import evaluate

sfr = evaluate.load("./metrics/script_fidelity_rate", module_type="metric")
sfr.compute(predictions=["کابل کې ښه هوا ده"], language="ps_af")

Hub metric after publishing:

import evaluate

sfr = evaluate.load("themechanism/script_fidelity_rate", module_type="metric")
sfr.compute(predictions=["کابل کې ښه هوا ده"], language="ps_af")

CLI

sfr score --language ps_af --text "کابل کې ښه هوا ده"
sfr audit predictions.jsonl --language ps_af --text-column prediction
sfr audit predictions.csv --language bn_in --text-column transcript --format csv

ASR batch example

from script_fidelity import compute_corpus_sfr

predictions = [
    item["text"]
    for item in whisper_outputs
]

summary = compute_corpus_sfr(predictions, language="bn_in")
print(summary["sfr_percent"])
print(summary["dominant_script_counts"])

pandas dataframe example

import pandas as pd
from script_fidelity import compute_sfr

df = pd.read_json("predictions.jsonl", lines=True)
df["sfr"] = df["prediction"].map(lambda text: compute_sfr(text, language="ps_af"))

Transformers compute_metrics example

import evaluate

wer = evaluate.load("wer")
sfr = evaluate.load("themechanism/script_fidelity_rate", module_type="metric")

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    pred_text = processor.batch_decode(predictions, skip_special_tokens=True)
    label_text = processor.batch_decode(labels, skip_special_tokens=True)
    return {
        "wer": wer.compute(predictions=pred_text, references=label_text),
        "sfr": sfr.compute(predictions=pred_text, language="ps_af")["sfr"],
    }

CI gate example

from script_fidelity import compute_corpus_sfr

summary = compute_corpus_sfr(predictions, language="ml_in")
if summary["sfr"] < 0.90:
    raise SystemExit("SFR regression: Malayalam output is below 90% target script")

shared-script caveats

SFR is a script check, not a language identifier. Pashto, Urdu, Persian, Arabic, Central Kurdish, and Sindhi share Arabic-script Unicode blocks. Latin-script languages mostly detect romanization or non-Latin substitution, not language identity. Pair SFR with language ID or lexical checks when shared-script confusions matter.

Use dominant_script() and script_distribution() to inspect failures:

from script_fidelity import dominant_script, script_distribution

dominant_script("this is romanized output")
script_distribution("বাংলা भाषा")

FLEURS codes

The registry covers the 102 FLEURS language configs listed by sfr languages. These paper languages have short aliases:

For the full reviewed registry, see script_fidelity/data/fleurs_registry.json.

Full code table: