tawfeer-llm

Arabic text is expensive to tokenize. This library measures exactly how expensive — and safely trims the part that isn't necessary.

pip install -e . from source · works with Claude, GPT, Gemini, or any LLM API · no model access needed

Why

Arabic uses more tokens per word than English for the same content on most tokenizers — higher cost, faster context exhaustion, slower responses. tawfeer-llm normalizes Arabic conservatively (never changes meaning) and tells you the exact, measured token savings — not a marketing estimate.

What it doesn't claim: normalizing Arabic does not make a model understand it better. This library's entire value case is cost, context size, and speed — each independently measurable, none of it guesswork.

Install

git clone https://github.com/AbdaullahAG/tawfeer-llm.git && cd tawfeer-llm
pip install -e ".[tokenizers]"

pip install -e ".[tokenizers,providers]"     # exact Anthropic/Gemini token counts
pip install -e ".[tokenizers,integrations]"  # LiteLLM / LangChain / LlamaIndex wrappers

First-time network note: the first call to report_savings() (or anything using get_default_counter()) downloads tiktoken's encoding file once, then caches it locally — no network needed after that. On a restrictive network, this first call can fail; see CONTRIBUTING.md for what that looks like and how to work around it.

Quick start

from ar_tokenwise import normalize, NormalizationLevel, report_savings

text = "مَرْحَـبًا بكم، اليوم ٢٠٢٦."
optimized = normalize(text, level=NormalizationLevel.LIGHT)

report = report_savings(text, optimized, cost_per_million_tokens=3.0)
print(report.tokens_saved, report.percent_saved, report.estimated_cost_savings_usd)

Before you normalize anything: Quranic/Hadith text, embedded ID numbers, and text displayed verbatim to a user should not be normalized — see skill/SKILL.md for the full, non-negotiable list.

What's included

Every item above is a heuristic where a heuristic is involved, and says so in its own docstring — see skill/REFERENCE.md for full usage and every documented limitation before relying on one in production.

How this compares

AraToken/aranizer require retraining or hosting your own model. PyArabic has no concept of LLM token cost. tawfeer-llm is middleware: drop-in normalization plus measurable reporting for any API you already call.

Benchmark

242 hand-authored sentences across MSA, four dialect regions, mixed Arabic-English, and formal/legal register — run it yourself:

python benchmark/run_benchmark.py

Full corpus methodology, a reusable quality-check tool, and honest caveats on every measured number: benchmark/README.md.

Docs

License

Apache License 2.0 — see LICENSE.