tawfeer-llm

Arabic text is expensive to tokenize. This library measures exactly how expensive — and safely trims the part that isn't necessary.

pip install ar-tokenwise[tokenizers] · works with Claude, GPT, Gemini, or any LLM API · no model access needed

Why

Arabic uses more tokens per word than English for the same content on most tokenizers — higher cost, faster context exhaustion, slower responses. tawfeer-llm normalizes Arabic conservatively (never changes meaning) and tells you the exact, measured token savings — not a marketing estimate.

What it doesn't claim: normalizing Arabic does not make a model understand it better. This library's entire value case is cost, context size, and speed — each independently measurable, none of it guesswork.

Install

pip install "ar-tokenwise[tokenizers]"

pip install "ar-tokenwise[tokenizers,providers]"     # exact Anthropic/Gemini token counts
pip install "ar-tokenwise[tokenizers,integrations]"  # LiteLLM / LangChain / LlamaIndex wrappers

First-time network note: the first call to report_savings() (or anything using get_default_counter()) downloads tiktoken's encoding file once, then caches it locally — no network needed after that. On a restrictive network, this first call can fail; see CONTRIBUTING.md for what that looks like and how to work around it.

Quick start

from ar_tokenwise import normalize, NormalizationLevel, report_savings

text = "مَرْحَـبًا بكم، اليوم ٢٠٢٦."
optimized = normalize(text, level=NormalizationLevel.LIGHT)

report = report_savings(text, optimized, cost_per_million_tokens=3.0)
print(report.tokens_saved, report.percent_saved, report.estimated_cost_savings_usd)

Preserve verified Quranic passages

Use preserve_quran=True when the input may contain Quranic quotations. The bundled reference corpus is loaded and indexed once per process; matching uses text with tashkeel and tatweel removed, while a confirmed passage is copied to the result exactly as the user supplied it. Four words are required for an independent match. Shorter two- or three-word portions are preserved only when they directly extend a confirmed quote.

from ar_tokenwise import normalize

text = "قال تعالى: بِسْمِ ٱللَّهِ ٱلرَّحْمَٰنِ ٱلرَّحِيمِ ٱلْحَمْدُ لِلَّهِ"
safe = normalize(text, preserve_quran=True)

The bundled source is Quran JSON; see QURAN_DATA_NOTICE.md for attribution. It remains a direct-text verifier, not a classifier: genuine spelling changes outside the configured comparison key will not match.

Before you normalize anything: Quranic/Hadith text, embedded ID numbers, and text displayed verbatim to a user should not be normalized — see skill/SKILL.md for the full, non-negotiable list.

What's included

Every item above is a heuristic where a heuristic is involved, and says so in its own docstring — see skill/REFERENCE.md for full usage and every documented limitation before relying on one in production.

How this compares

AraToken/aranizer require retraining or hosting your own model. PyArabic has no concept of LLM token cost. tawfeer-llm is middleware: drop-in normalization plus measurable reporting for any API you already call.

Benchmark

242 hand-authored sentences across MSA, four dialect regions, mixed Arabic-English, and formal/legal register — run it yourself:

python benchmark/run_benchmark.py

Full corpus methodology, a reusable quality-check tool, and honest caveats on every measured number: benchmark/README.md.

Docs

License

Apache License 2.0 — see LICENSE.