WeTextProcessing Runtime
Multilingual text normalization for speech applications, without a Pynini runtime dependency.
Quick start · Languages · Python API · Streaming · Development
wetext runs the weighted finite-state transducer (WFST) grammars from
WeTextProcessing using
kaldifst. Published wheels bundle the compiled
models; Pynini is needed only when building models from source.
- TN and ITN — convert written expressions into spoken forms for speech synthesis, or spoken expressions into written forms after speech recognition.
- Four languages — Chinese, English, Japanese, and Korean through one Python API and CLI.
- Inspectable results — unique n-best candidates, WFST costs, and exact input/output token spans.
- Streaming input — feed new text chunks and finalize each utterance with
flush(). - On-demand models — load and cache only the languages and processing options your application uses.
Supported languages
| Language | Code | TN | ITN | Example |
|---|---|---|---|---|
| Chinese | zh |
Yes | Yes | 12 → 十二 |
| English | en |
Yes | Yes | 12 → twelve |
| Japanese | ja |
Yes | Yes | 12 → 十二 |
| Korean | ko |
Yes | Yes | 12 → 십이 |
lang="auto" detects Hangul, Japanese kana, and Chinese characters. Digits alone
use Chinese; Latin text uses English. Specify a language for ambiguous inputs,
including Japanese text containing only kanji and numbers.
Korean support is available in this source checkout. Before the next package release, follow the model build instructions.
Quick start
Install
pip install wetext
Python
from wetext import Normalizer
normalizer = Normalizer(lang="zh", operator="tn")
print(normalizer.normalize("价格是12.5元")) # 价格是十二点五元
inverse_normalizer = Normalizer(lang="en", operator="itn")
print(inverse_normalizer.normalize("twenty three dollars and fifty cents")) # $23.50
Command line
wetext --lang zh --operator tn "价格是12.5元"
wetext --lang en --operator itn "twenty three dollars and fifty cents"
wetext --lang ja --operator tn "12時"
wetext --lang ko --operator tn "오후 3시 30분"
wetext --lang ko --operator itn "오천원"
Each command prints the normalized text. --lang defaults to auto and
--operator defaults to tn. Inspect all options with wetext --help.
Python API
Normalize text
Choose the language and direction when constructing a Normalizer. Individual
calls can override the same options:
from wetext import Normalizer
normalizer = Normalizer(lang="zh", operator="tn", remove_erhua=True)
print(normalizer.normalize("全新版本儿,简直666")) # 全新版本,简直六百六十六
print(normalizer.normalize("十二", operator="itn")) # 12
The function API is also available as wetext.normalize(text, lang="zh", operator="tn").
Input is trimmed before normalization. Korean TN retains meaningful trailing
spaces emitted by upstream rules:
from wetext import Normalizer
normalizer = Normalizer(lang="ko", operator="tn")
print(repr(normalizer.normalize("2024년"))) # '이천이십사년 '
print(normalizer.normalize("십이", operator="itn")) # 12
Processing options
All Boolean options default to False. The CLI uses the corresponding
hyphenated flag, for example remove_erhua=True becomes --remove-erhua.
| Python option | Behavior |
|---|---|
fix_contractions |
Expand English contractions, for example don't → do not |
traditional_to_simple |
Convert traditional Chinese characters to simplified Chinese |
full_to_half |
Convert full-width characters to half-width |
remove_interjections |
Apply the Chinese interjection-removal rules |
remove_puncts |
Remove punctuation |
tag_oov |
Apply the Chinese vocabulary's out-of-vocabulary tagging rules |
enable_0_to_9 |
Enable single-digit conversion for Chinese and Japanese ITN |
remove_erhua |
Remove erhua suffixes for Chinese TN |
wetext --traditional-to-simple "你好,這是測試。"
wetext --remove-puncts "你好,這是測試。"
wetext --lang zh --remove-erhua "全新版本儿"
N-best candidates and costs
Request up to nbest distinct outputs, ordered by WFST cost:
from wetext import Normalizer
normalizer = Normalizer(lang="en", operator="tn")
print(normalizer.normalize("4x6", nbest=3))
# ['four by six', 'four times six', 'four x six']
candidates = normalizer.normalize_candidates("4x6", nbest=3)
print(candidates[0].text, candidates[0].cost)
Costs are tropical WFST weights: lower is better; they are not probabilities.
normalize() returns a string for nbest=1, or a list for nbest>1.
normalize_candidates() always returns a list with text, costs, and path ranks.
Exact input/output mappings
from wetext import Normalizer
normalizer = Normalizer(lang="zh", operator="itn")
result = normalizer.normalize_with_mapping("价格是十三点五元")
print(result.output_text) # 价格是¥13.5
for mapping in result.mappings:
print(mapping.token_type, mapping.input_text, "=>", mapping.output_text)
# money 十三点五元 => ¥13.5
Mappings trace the selected WFST path. Offsets are half-open Unicode character
positions in the original input and normalized output. Use include_identity=True
to include unchanged tokens, or result.as_dict() for a serializable representation.
The mapping API returns a single NormalizationResult for nbest=1 and a list
for nbest>1. It cannot be combined with fix_contractions=True, because
contraction expansion is not represented by the WFST path.
Streaming
Pass only the new text to each feed() call. The return value is the complete
current output, which may change while a trailing expression is unfinished.
Use one instance per utterance and call flush() when it ends:
from wetext import StreamNormalizer
stream = StreamNormalizer(lang="zh", operator="itn")
print(stream.feed("今天是二零")) # 今天是二零
print(stream.feed("二六年")) # 今天是2026年
print(stream.flush()) # 今天是2026年
stream = StreamNormalizer(lang="zh", operator="tn")
print(stream.feed("价格是12")) # 价格是十二
print(stream.feed(".5元")) # 价格是十二点五元
print(stream.flush()) # 价格是十二点五元
Streaming requires an explicit language. Its default direction is itn;
Normalizer defaults to tn. Call reset() before reusing a flushed stream.
Korean retains its input context until flush() to preserve exact normalization.
Older model bundles without streaming prefix graphs also retain more context.
For unusually long unfinished expressions, only the tail is normalized for the
current display; flush() normalizes all remaining input exactly.
Development and model builds
The runtime and grammar compiler have separate dependencies:
| Task | Requirements |
|---|---|
| Run a published wheel | wetext and its runtime dependencies; bundled FST models |
| Build models | WeTextProcessing source and a working Pynini/OpenFst environment |
| Run source tests | Generated FST models and pytest |
To build all models from the upstream revision used by this release workflow:
git clone https://github.com/wenet-e2e/WeTextProcessing.git /tmp/WeTextProcessing
git -C /tmp/WeTextProcessing checkout 31779ac
pip install -e . /tmp/WeTextProcessing pytest build
PYTHONPATH=/tmp/WeTextProcessing python build_fsts.py
python -m pytest tests -q
python -m build
Pynini must be available before building. If no compatible wheel is available, follow the Pynini installation instructions.
Generated models live in wetext/fsts/ and are included in distribution packages.
Rebuild a single language with python build_fsts.py --lang ko (with the same
upstream PYTHONPATH). Korean models require upstream commit 31779ac or later,
which includes WeTextProcessing PR #381.
The release workflow accepts an upstream_ref to select the grammar revision and
runs regression tests before packaging. Models are loaded on first use and
cached in process; the runtime does not compile rules or write a model cache.
License
Metadata
Release files for wetext 0.1.9
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| wetext-0.1.9.tar.gz | 5.1 MB | Without provenance | Details for wetext-0.1.9.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| wetext-0.1.9-py3-none-any.whl | Python 3 | none | any | Details for wetext-0.1.9-py3-none-any.whl |
Total release size: 10.4 MB
Release files / wetext-0.1.9.tar.gz
| Download URL | wetext-0.1.9.tar.gz |
|---|---|
| Size | 5.1 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8bc16e9a4d29d2f1411e69c829a166541581ab0c7764dd0c63357ee4e76578da
|
|
BLAKE2b-256 checksum How to use checksums |
a2466a732d669308035cf92e08c3636dffcefc2ac770bbaed045e69a8c9ffc0d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.17
|
Release files / wetext-0.1.9-py3-none-any.whl
| Download URL | wetext-0.1.9-py3-none-any.whl |
|---|---|
| Size | 5.3 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
5ead0770f2e82f93966852fcb563390985fa19575e0a15c53168dfc331e356de
|
|
BLAKE2b-256 checksum How to use checksums |
4768863ef63815d13b5930870b464bc7eaa848b8c7c0d3545a2a065bcd288193
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.17
|
Security for wetext 0.1.9
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.