This release is a pre-release and may not be stable for production use.
arbtok
Rule-based Arabic (MSA) text→IPA with tashkeel diacritization — a downstream Arabic engine built on orthography2ipa.
Word phonology is built on the orthography2ipa shared lattice: the
language-agnostic grapheme tokenizer (PhonetokTokenizer) over the ar
spec grapheme table produces a per-position candidate lattice. The ar
engine (orthography2ipa ≥ 1.64) handles the segment-local phonology
natively — gemination (shadda ّ, glides included), lam-alif / presentation
ligatures (ﻻ → laː), onset glides (يَ → ja), a hamza carrier's bare
/ʔ/ before an explicit harakah, the word-final glide as a long vowel, and
pausal tāʾ marbūṭa. The Arabic morpho-phonology that the shared table still
cannot express is layered on as composable LatticeRescorers
(arbtok/lattice.py) rather than a private tokenizer fork:
- sun-letter assimilation (idghām ash-shamsiyya) — the lām of the
definite article ⟨ال⟩ assimilates into a following coronal (sun) letter
(
al-šams→aš-šams); moon letters keep the lām (al-qamar); - hamzat al-waṣl elision — a word-initial prosthetic alif is silent,
its harakah carrying the vowel (
istiqbāl); - accusative-alif silencing after tanwīn al-fatḥ (
marħaban), mater-lectionis length for a fatḥa + standalone alif maksūra (ħattaː), a sukūn-final coda glide (ظَبْي →ðˤabj), and the bare glottal stop of a hamza carrier before a sukūn or word edge (taʔθīr).
Emphatic (pharyngealization) spreading rides on the ar spec's own B8
allophone_rules. Cross-word sandhi — clitic joining, cross-word waṣl
elision, tanwīn pausal forms, tāʾ marbūṭa, and idgham/iqlab nasal
assimilation — is orthogonal to the word lattice and lives in the
sentence-level orchestration. Bare (undiacritized) text is diacritized
first via text2tashkeel —
a model picker over bundled ONNX diacritization models.
Honesty note: the gold IPA reference set was LLM-generated and has not been validated by a native MSA speaker. If you speak MSA, pull requests are very welcome.
Installation
pip install arbtok
Usage
arbtok is built on orthography2ipa
(spec data and the shared G2PPlugin/WordContext base types) and owns the
Arabic pipeline — orthography2ipa stays the language-agnostic base library.
Engine class
from arbtok.tokenizer import Sentence
Sentence("اَلسَّلَامُ عَلَيْكُمْ").ipa
An isolated MSA word transcribes on the shared lattice directly:
from arbtok.lattice import word_ipa
word_ipa("الشَّمْس") # 'aʃʃams' — sun-letter assimilation as a rescorer
word_ipa("الْقَمَر") # 'alqamar' — moon-letter control (lām kept)
Bare text is handled by diacritizing first:
from arbtok.plugin import ArbtokG2PPlugin
plugin = ArbtokG2PPlugin()
plugin.transcribe("كتاب جميل") # auto-tashkeel + IPA
Diacritization only
from arbtok.tashkeel import TashkeelDiacritizer # wraps text2tashkeel
TashkeelDiacritizer().diacritize("كتاب جميل")
Quality benchmarks
The test suite pins a gold sentence set (CER target ≤ 5% against the
reference transcriptions) and benchmarks against espeak-ng. See
tests/test_ipa_fuzzy.py and docs/ for details.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file arbtok-0.0.0a6.tar.gz.
File metadata
- Download URL: arbtok-0.0.0a6.tar.gz
- Upload date:
- Size: 122.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
28d7b78dafce43ecb4f05ef8775b4306a33988f82d2609ca26766316c614dd7c
|
|
| MD5 |
abb08d9bdaf24e3d64b92555b3f7f81a
|
|
| BLAKE2b-256 |
4c556fb477aa02af585b6ce62bcc16468d3519704ce076ea723d8fd869ed107d
|
File details
Details for the file arbtok-0.0.0a6-py3-none-any.whl.
File metadata
- Download URL: arbtok-0.0.0a6-py3-none-any.whl
- Upload date:
- Size: 114.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
74b621fd38075f637c3caa3e4fd64b0b268b787e8dffb3207636edde4ec2eaa8
|
|
| MD5 |
b934aad19fcbac6eabf898fb9a8980d3
|
|
| BLAKE2b-256 |
d862a3624d05a767e01d7e2d0a5cb9e62d4e401df4d8e626104739708ef2bf62
|
