Skip to main content

Sindhi Tokenizer: A Corpus-Driven SentencePiece Unigram Tokenizer

Sindhi Tokenizer

Python 3.9+ License: MIT Hugging Face Models Documentation Tests Round-trip check: 0.000% normalized-length loss

A subword tokenizer purpose-built for the Sindhi language (سنڌي) in the Perso-Arabic script. Trained with SentencePiece Unigram with full observed-character coverage and byte-level fallback on 1.46M+ sentences. It was benchmarked against 14 baseline tokenizers across four evaluation registers. The benchmark has 18 entries: 14 baseline entries and four custom vocabulary sizes. Qwen2.5 and SindhiLM_v2 are identical in this evaluation, so the entries represent 17 distinct tokenizer behaviors. The round-trip check reports 0.0000% normalized-length loss on evaluated text; this does not establish exact identity for arbitrary input.



Table of Contents

  1. Project Overview
  2. Motivation and Problem Statement
  3. Project Objectives and Scope
  4. Key Features
  5. Technology Stack
  6. Repository Architecture
  7. Architecture and Workflow Diagrams
  8. Dataset Documentation
  9. Tokenizer Methodology
  10. Training Pipeline & Negative Results
  11. Installation and Usage
  12. Evaluation and Comprehensive Benchmarks
  13. Linguistic & Morpheme Analysis
  14. Downstream NLP Applications
  15. Limitations and Known Issues
  16. Complete Documentation Index
  17. References and Further Reading
  18. Team and Acknowledgments
  19. License and Citation
  20. Frequently Asked Questions (FAQ)

Project Overview

Sindhi is an Indo-Aryan language spoken by an estimated 25–30 million people, predominantly in the Sindh province of Pakistan and parts of India. Despite its rich literary heritage, Sindhi remains a severely under-resourced language in Natural Language Processing (NLP).

Existing modern Large Language Models (LLMs) and multilingual tokenizers (such as XLM-RoBERTa, LLaMA, Qwen, and BLOOM) under-represent Sindhi orthography, causing high token fertility, extreme sequence bloat, out-of-vocabulary (UNK) fallbacks, or destructive character drops.

This repository provides:

  • Four Trained Models: Pretrained SentencePiece Unigram models at 32k, 64k (flagship recommended), 100k, and 150k vocabulary sizes.
  • Corpus Engineering Pipeline: A 1.46M+ sentence corpus drawn from Sindhi and regional Wikipedia, mostly news text, a small Twitter component, and an Urdu/Persian/Arabic anchor that makes up 16.1% of the corpus by character count (89,449 unique sentences, repeated four times).
  • Full-Scale Benchmark Suite: Standardized evaluation of 18 tokenizers across 4 distinct registers (FLORES+, Wikipedia, Sentiment, Sindh Line News).
  • Round-Trip Length Check: A normalized-length check that can flag text shortening; targeted examples were also inspected for deleted characters.
  • Interactive UI: A Gradio web application for real-time subword tokenization inspection.

Motivation and Problem Statement

1. Unique Script and Extended Perso-Arabic Orthography

Sindhi uses a 52-letter extended Perso-Arabic alphabet containing implosive consonants (ٻ, ڄ, ڏ, ڳ), aspirated digraphs (ڀ, ٿ, ڇ, ٺ), retroflex consonants (including ڻ), and specialized diacritics. General Arabic or Urdu tokenizers frequently split these single graphemes into multi-byte fallback sequences, dramatically inflating token counts.

2. The Token Fertility Tax

When a tokenizer splits a single word into 3–5 subword fragments, context window efficiency drops proportionally. A 2048-token context window accommodates only ~707 Sindhi words under LLaMA 3.2 (fertility 2.896) or ~1,354 words under XLM-RoBERTa (1.512), compared to ~1,786 Sindhi words under CustomSindhi_64k (1.147).

3. Silent Character Deletion in Competing Models

Standard tokenizer benchmarks often report token fertility and UNK rate without checking decoded output. During this project's audit, a normalized-length round-trip check, followed by inspection of affected examples, surfaced text-shortening defects in existing tokenizers:

  • TinyLlama_Sindhi: Drops native Sindhi letters such as U+06C1 (HEH GOAL) and corrupts text encoded with Arabic Presentation Forms, destroying data in up to 24.9% of sentences.
  • FahadQazi_BPE: Silently deletes Latin characters in code-switched technical text, corrupting up to 16.2% of sentences.
  • Our model: Shows 0.0000% normalized-length loss under this check on evaluated text with byte-level fallback; the metric is not proof of exact character identity.

Project Objectives and Scope

Primary Objectives

  1. Deliver Flagship Tokenizer: Train and release a high-efficiency Sindhi subword tokenizer achieving the lowest fertility and highest single-token preservation across diverse registers.
  2. Measure Round-Trip Integrity: Use byte fallback and full observed-character coverage, then report the limits of the normalized-length check.
  3. Rigorous Empirical Comparison: Benchmark 14 baseline tokenizers under identical, unsampled test conditions.
  4. Transparent Documentation: Document negative results (what didn't work), corpus preparation heuristics, and evaluation caveats.

Supported Use Cases

  • Large Language Model (LLM) pretraining and fine-tuning for Sindhi.
  • Sequence classification (Sentiment Analysis, Topic Modeling).
  • Machine Translation (FLORES+ benchmarks).
  • Named Entity Recognition (NER) and Information Retrieval.

Known Exclusions

  • Speech recognition (ASR) acoustic unit modeling.
  • Devnagari script Sindhi (this release targets Arabic-script Sindhi).

Key Features

  • SentencePiece Unigram Architecture: Statistically models subword probabilities using maximum likelihood estimation, proven superior to standard greedy BPE for morphologically rich languages.
  • 100% Character Coverage + Byte Fallback: Zero unknown (<unk>) token emissions. Unknown Unicode characters decompose cleanly into raw UTF-8 byte tokens without data corruption.
  • Flagship 64k Model: Balances vocabulary efficiency (1.147 fertility) with embedding table size, yielding an 88.6% single-token word retention rate.
  • Round-Trip Check Result: 0.0000% normalized-length loss across the evaluated corpora; this does not establish exact character identity.
  • Clean Subword Boundaries: Preserves leading whitespace symbols ( , \u2581) for precise detokenization without text corruption.

Technology Stack

Layer Technologies Purpose
Core Engine Python 3.9+, SentencePiece (C++ bindings) Tokenizer training, Viterbi segmentation, byte fallback
NLP Framework Hugging Face Transformers, huggingface_hub Adapter integration, model sharing and distribution
Datasets Hugging Face Datasets, Pandas, NumPy Corpus ingestion, normalization, filtering, metrics computation
Interactive Demo Gradio Real-time interactive subword visualization UI
Quality & CI/CD Pytest, GitHub Actions Automated regression tests across Python 3.9–3.12
Documentation GitHub Pages, Markdown, Mermaid.js Static web docs, draft research manuscript, technical explainer

Repository Architecture

sindhi-tokenizer/
├── .github/
│   └── workflows/
│       └── python-tests.yml        # Automated CI workflow testing Python 3.9-3.12
├── .gitignore                      # Excludes models, artifacts, cache, and virtualenvs
├── CITATION.cff                    # Citation File Format (v1.2.0)
├── DATA_SOURCES.md                 # Complete provenance, volume, and licensing documentation
├── LICENSE                         # MIT Open Source License
├── README.md                       # Central documentation index and project guide
├── pyproject.toml                  # PEP 517/621 package metadata and dependency configuration
├── requirements.txt                # Exact runtime dependencies
├── demo/
│   ├── README.md                   # Instructions for running and deploying the demo
│   └── app.py                      # Interactive Gradio comparison web app
├── docs/
│   ├── paper.md                    # Draft research manuscript, not yet published (14-baseline study)
│   └── explainer.md                # 21-page technical companion log and negative results
├── evaluation/
│   ├── README.md                   # Evaluation metrics guide and CLI/API documentation
│   ├── __init__.py                 # Package initialization
│   └── metrics.py                  # Fertility, single-token, UNK, byte-fallback, and fidelity engine
├── notebooks/
│   ├── README.md                   # Reproducible notebook guide
│   ├── training.ipynb              # Kaggle/Jupyter notebook: corpus download, split & training
│   └── benchmark.ipynb             # Kaggle/Jupyter notebook: 18-tokenizer evaluation suite
├── results/
│   └── README.md                   # Benchmark tables and summary report across 4 registers
└── training/
    ├── README.md                   # Training recipes, regex splitters, and negative results
    ├── __init__.py                 # Package initialization
    └── train_tokenizer.py          # Standalone training script with optimal SentencePiece parameters

Architecture and Workflow Diagrams

1. End-to-End System Architecture

flowchart TD
    subgraph Data_Pipeline["1. Corpus Assembly & Preprocessing"]
        A1["Sindhi Wikipedia<br/>(6,000 articles)"] --> P["Sentence Extraction & Cleaning<br/>Regex Split: [۔.!\n؟]+<br/>Drop fragments of 3 characters or fewer"]
        A2["Sindhi-NLP Dataset<br/>(Twitter + Kawish + Awami Awaz)"] --> P
        A3["shaikhsalman Corpus<br/>(20k rows, no language filter)"] --> P
        A4["Multilingual Regional Anchor<br/>(Urdu, Persian, Arabic, repeated ×4, 16.1% of characters)"] --> P
        P --> C["Balanced Training Corpus<br/>(1.46M+ Sentences)"]
    end

    subgraph Training_Engine["2. Model Training (SentencePiece)"]
        C --> SP["SentencePiece Unigram Trainer<br/>character_coverage=1.0<br/>byte_fallback=True<br/>split_by_whitespace=True"]
        SP --> M1["Custom 32k Model"]
        SP --> M2["Custom 64k Model (Flagship)"]
        SP --> M3["Custom 100k Model"]
        SP --> M4["Custom 150k Model"]
    end

    subgraph Evaluation_Suite["3. Rigorous Evaluation Engine"]
        M2 --> E["evaluation/metrics.py"]
        REG1["FLORES+ (snd_Arab)"] --> E
        REG2["Wikipedia Formal (25.4k sent)"] --> E
        REG3["Sentiment Informal (3.5k sent)"] --> E
        REG4["Sindh Line News (87.1k sent)"] --> E
        E --> MET["Metrics Computed:<br/>• Token Fertility<br/>• Single-Token Word %<br/>• UNK Rate (0.000%)<br/>• Byte Fallback Rate<br/>• Normalized-Length Loss (0.000%)"]
    end

    subgraph Downstream["4. Downstream Deployment"]
        M2 --> HF["Hugging Face Hub<br/>alinawazmahar/sindhi-tokenizer"]
        M2 --> GR["Interactive Gradio Demo<br/>demo/app.py"]
        M2 --> LLM["Sindhi LLM Pretraining & NLP"]
    end

2. Text Encoding and Round-Trip Fidelity Flow

sequenceDiagram
    autonumber
    participant Input as Raw Sindhi Text
    participant Norm as NFKC & Formatting Normalizer
    participant Tokenizer as SPMTokenizerAdapter
    participant Tokens as Subword Tokens / Byte IDs
    participant Output as Decoded Reconstructed Text
    participant Audit as Normalized-Length Check

    Input->>Norm: Normalize Unicode (NFKC) & remove zero-width formatting
    Norm->>Tokenizer: Clean text string
    Tokenizer->>Tokens: Encode via Unigram Viterbi path (out_type=int)
    Tokens->>Output: Decode token IDs back to string
    Output->>Audit: Compare Original vs Decoded
    Note over Audit: Loss = max(0, input_length - decoded_length) / input_length<br/>Custom Model Result: 0.0000% normalized-length loss

Dataset Documentation

Training and evaluation records are separated where known, but the sentiment evaluation set shares publisher-adjacent provenance with a training source. See DATA_SOURCES.md for source details and licensing status.

Training Corpus (1.46M+ Sentences)

Source Dataset Origin / Provider Volume Used Preprocessing & Filtering Rules License
Sindhi Wikipedia wikimedia/wikipedia (20231101.sd) 6,000 articles Articles 2,001–8,000 (first 2,000 reserved for evaluation). Fragments of 3 characters or fewer dropped; no word-count limit (the 3–100 word filter is applied only to the evaluation set). CC BY-SA 3.0
Sindhi-NLP GitHub: Sindhi-NLP/sindhi-NLP-dataset 4 CSV collections Primarily news (98.4% Awami Awaz and Daily Kawish articles) with 1.6% social tweets. MIT
shaikhsalman Corpus shaikhsalman/sindhi-llm-corpus 20,000 rows Seed 7 shuffle; 630,180 sentences, about 57% of the Sindhi-side corpus. No language filter was applied. A diagnostic found about 11.3% of sampled sentences were mostly Latin script, and they remain in the training text. Project owner reports free use; exact license and conditions are not shown on the card (see DATA_SOURCES.md)
Multilingual Anchor Urdu, Persian, Arabic Wikipedia 1,500 articles each 89,449 unique sentences (4.9M characters) repeated four times to reach 16.1% of total corpus characters (target was 15%). Exposes model to shared Perso-Arabic roots. CC BY-SA 3.0

Evaluation Registers (Unsampled Benchmarks)

Register Dataset Identifier Sentences Domain / Linguistic Style
flores200_anchor openlanguagedata/flores_plus (snd_Arab) 1,012 Standardized translation benchmark; formal and modern register.
wikipedia_formal wikimedia/wikipedia (20231101.sd) 25,480 Formal encyclopedic Sindhi prose (first 2,000 articles, disjoint from train).
sentiment_informal (legacy ID) alinawazmahar/Sindhi_Sentiment_dataset 3,536 Newspaper text pseudo-labeled for sentiment; this is news, not social-media text.
sindh_line_news Mozilla Data Collective (Sindh Line Publishers) 87,158 Contemporary, daily regional news coverage (CC BY-SA 4.0).

Tokenizer Methodology

Why SentencePiece Unigram over Greedy BPE?

Byte-Pair Encoding (BPE) merges the most frequent character pairs iteratively. In morphologically complex, agglutinative, or clitic-heavy languages like Sindhi, greedy merges frequently fuse grammatical suffixes onto root stems incorrectly.

The Unigram language model algorithm (Kudo, 2018) initializes an oversized vocabulary and optimizes subword segmentation using maximum likelihood:

$$P(X) = \max_{\mathbf{x} \in S(X)} \prod_{i=1}^{M} P(x_i)$$

This statistically discovers true morphological subwords while optimizing whole-sequence likelihood.

Step-by-Step Tokenization Example

Consider the authentic Sindhi phrase: $$\text{خداوند ڪريم کيس اعليٰ ذهانت سان نوازيو هو}$$ (The Almighty Lord had blessed him with high intellect)

  1. Input Normalization: NFKC normalization ensures canonical representation of Arabic script forms.
  2. Whitespace Prefixing: Words receive the boundary marker (\u2581).
  3. Unigram Segmentation:
    [' خداوند', ' ڪريم', ' کيس', ' اعليٰ', ' ذهانت', ' سان', ' نوازيو', ' هو']
    
  4. Token ID Mapping:
    [12450, 4812, 1089, 7543, 14210, 521, 18920, 412]
    
  5. Round-trip check: Evaluated text showed 0.0000% normalized-length loss. This length-based check can miss equal-length substitutions or reorderings.

Training Pipeline & Negative Results

1. Training Parameters (training/train_tokenizer.py)

python training/train_tokenizer.py \
  --corpus corpus.txt \
  --vocab_size 64000 \
  --model_prefix models/sindhi_custom_64000
  • model_type="unigram"
  • character_coverage=1.0 (Ensures every single observed Unicode character is covered)
  • byte_fallback=True (Unknown characters fall back to raw byte tokens <0xXX>)
  • split_by_whitespace=True
  • unk_id=0, bos_id=1, eos_id=2, pad_id=-1

2. Documented Negative Results (What Failed and Why)

In accordance with rigorous research standards, two negative experimental results are preserved in docs/explainer.md:

  1. Forced Postposition Tokens: Forcing common postpositions (e.g., جو, جي, کي) and loanwords to remain single tokens degraded the training objective (final objective value rose from ≈9.6 unforced to ≈14.9 forced at 32k vocab). Given enough data, Unigram naturally discovers these tokens with superior probability allocations.
  2. Pre-marking Clitic Boundaries: Pre-segmenting supposed fused clitics with whitespace proved harmful. Investigation revealed that suspected "fused forms" were simply coincidental orthographic overlaps with unrelated lexical roots.

Installation and Usage

Install the Python SDK

Requires Python 3.9 or newer. The SDK only installs SentencePiece and the Hugging Face Hub client; training and demo dependencies are optional.

pip install sindhi-tokenizer

Quickstart

from sindhi_tokenizer import SindhiTokenizer

# Downloads the recommended 64k model once, then reuses the Hub cache.
tokenizer = SindhiTokenizer.from_pretrained()
text = "سنڌ جي تاريخ تمام قديم آهي"

pieces = tokenizer.tokenize(text)
token_ids = tokenizer.encode(text)

print("Pieces:", pieces)
print("Token IDs:", token_ids)
print("Decoded:", tokenizer.decode(token_ids))
print("Vocabulary size:", tokenizer.vocab_size)  # 64000

Choose another released vocabulary with vocab_size=32000, 100000, or 150000. You can also load a local .model file with SindhiTokenizer.from_file(path). Batch methods are encode_batch() and decode_batch().

To work on the repository and its training/demo tools:

git clone https://github.com/alinawazcode/sindhi-tokenizer.git
cd sindhi-tokenizer
pip install -e ".[dev,training,demo]"

The model download is cached by Hugging Face Hub. The SDK code is MIT licensed; the model files are separately licensed under CC BY-SA 4.0. Review the model card for attribution and sharing terms before redistributing model files.


Evaluation and Comprehensive Benchmarks

All 18 benchmark entries were evaluated across the four evaluation registers without subsampling. SindhiLM_v2 is identical to Qwen2.5 for this evaluation; the entries therefore represent 17 distinct tokenizer behaviors.

Macro Benchmark Summary (Unweighted Mean across 4 Registers)

Tokenizer Vocab Size Fertility (tokens/word) ↓ Single-Token Word % ↑ UNK Rate % ↓ Byte-Fallback % ↓ Worst Normalized-Length Loss % ↓
CustomSindhi_150k 150,000 1.125 89.7% 0.000% 0.154% 0.0000%
CustomSindhi_100k 100,000 1.133 89.3% 0.000% 0.149% 0.0000%
CustomSindhi_64k (Flagship) 64,000 1.147 88.6% 0.000% 0.138% 0.0000%
CustomSindhi_32k 32,000 1.184 86.5% 0.000% 0.113% 0.0000%
TinyLlama_Sindhi 42,681 1.208 84.4% 0.000%* 0.000%* 0.4147% (Corrupts 24.9% sent)
hellosindh_SindhiBERT_NER 32,001 1.218 83.7% 0.194% 0.197% 0.0751%
mBERT_Sindhi_Kashif786 119,547 1.278 78.4% 0.143% 0.000% 0.0021%
XLM-RoBERTa 250,002 1.512 65.4% 0.034% 1.402% 0.0040%
SindhiLM_v3 32,000 1.730 55.8% 3.605% 0.000% 0.0754%
FahadQazi_BPE 13,031 1.919 39.1% 0.000%* 7.127%* 0.8461% (Corrupts 16.2% sent)
hellosindh_SindhiBERT 32,000 1.960 35.5% 0.122% 0.114% 0.0089%
mT5 250,100 2.280 21.7% 0.022% 12.746% 0.0040%
Gemma3 262,144 2.308 26.8% 0.000% 0.105% 0.0000%
Qwen2.5 / SindhiLM_v2 151,643 2.580 23.8% 0.000% 8.665% 0.0000%
BLOOM 250,680 2.627 24.7% 0.000% 31.660% 0.0000%
Llama3.2 128,000 2.896 18.2% 0.000% 28.472% 0.0000%
SindhiLM_v1 43 4.306 0.1% 14.017% 0.000% 0.0790%

* TinyLlama_Sindhi and FahadQazi_BPE show apparent 0.000% UNK rates because both models silently delete unsupported characters rather than emitting a UNK token. FahadQazi_BPE shows real, severe character loss on round-trip fidelity testing alongside high byte-fallback. SindhiLM_v2 is numerically identical to Qwen2.5 across all metrics; both are listed in the table for completeness.

Evaluation Takeaways

  1. Lowest fertility and highest single-token retention at ≥64k: CustomSindhi (64k, 100k, 150k) has the lowest fertility and highest single-token retention in all four registers. The margin on FLORES+ at 64k is 0.002 fertility and 0.3 points of single-token retention, which we treat as parity; no significance tests were run. At 32k, one narrow exception exists: TinyLlama_Sindhi beats CustomSindhi_32k on FLORES+ fertility (though it carries a critical character-deletion defect). On byte-fallback, CustomSindhi_64k (0.138%) achieves near-lowest fallback, with Gemma 3 slightly lower at 0.105%.
  2. Efficiency gains at 64k: Mean fertility improves by 24.1% over XLM-RoBERTa and 60.4% over LLaMA 3.2. This describes fertility and single-token retention, not every metric.
  3. Round-trip check results: Custom models show 0.0000% normalized-length loss under the stated check on all four registers. This length-based check does not prove exact character identity; TinyLlama_Sindhi and FahadQazi_BPE show measurable length loss.

Linguistic & Morpheme Analysis

To verify that vocabulary expansion did not simply overfit high-frequency substrings, all tokenizers were evaluated on a targeted 22-word linguistic test suite spanning four distinct categories:

  1. Genitive & Possessive Constructions (دنيا جو, پاڻي جي, ماڻهن جو): All Custom models split these cleanly into base noun + postposition clitic.
  2. Inflectional Suffixes (ڪتابن, شهرن, توهانکي, منهنجو): Preserved as single tokens in Custom 64k models.
  3. Sindhi Implosives & Aspirated Digraphs (ٻار, ڄاڻ, ڏينهن, ڳوٺ, ڀاءُ, ٿورو, ڇوڪرو): Never broken into disconnected diacritics or fallback bytes.
  4. Loanwords & Technical Terms (ڪمپيوٽر, انٽرنيٽ, اسڪول, تعليم): Retained as intact single lexical units.

Intended Downstream NLP Applications & Hypotheses

Note: As documented in the paper (§6), the tokenizer is designed for textual NLP representation; downstream task-specific fine-tuning (e.g. BLEU/ROUGE benchmarks) represents intended future work.

Intended Application Hypothesized Architectural Benefit
Sindhi Large Language Models (LLMs) Decreases token sequence length by 50–60% vs general multilingual LLMs (LLaMA, BLOOM), effectively doubling or tripling context length capacity during pretraining and inference.
Machine Translation (MT) Preserves postpositions and implosive graphemes intact, intended to reduce subword fragmentation and hallucination during sequence-to-sequence translation.
Sentiment Analysis & Text Classification Retains full grammatical morphemes, yielding richer dense embeddings for subword encoders.
Named Entity Recognition (NER) Eliminates character drop bugs that corrupt proper nouns, locations, and transliterated person names.

Limitations and Known Issues

In accordance with rigorous academic transparency (see docs/paper.md §6), ten limitations are explicitly recorded:

  1. Absence of Classical / Literary Register: No classical or literary Sindhi register is included. Five independent candidate sources were investigated: a corrupted CSV export, an unrelated SQL dump, two unusable OCR scans (one incompatibly licensed), and a government-affiliated platform gated behind an institutional subscription.
  2. Partial Source Overlap in Evaluation: The Sentiment register and the Sindhi-NLP component of the training corpus share adjacent news-source provenance; full genre-level independence is not established for this pair.
  3. Metric-specific results: Against FLORES+ specifically, CustomSindhi_32k has one confirmed loss on fertility and single-token retention. The all-register lead at 64k and above refers to those two measures; byte-fallback is a separate trade-off.
  4. Qualitative Test Does Not Differentiate at Word Level: On the 22-word test list, the new tokenizer ties rather than beats the best competitor on 21 of 22 words; the demonstrated advantage is aggregate across full corpora, not example-level.
  5. Search for Existing Tokenizers May Be Incomplete: Four additional tokenizers were discovered after initial benchmarking; while subsequent searches found no further models, the open-source NLP space is active enough that this cannot be guaranteed exhaustive.
  6. Downstream Applications Undetermined: The tokenizer's target use is future Sindhi language modeling. It is not validated for Automatic Speech Recognition (ASR) or Text-to-Speech (TTS).
  7. Register Weighting Choice: The four evaluation registers differ substantially in size; an unweighted macro mean was used rather than a token-pooled mean, which would have been dominated by Sindh Line News (~2.5× the other three combined).
  8. No language filter on training text: About 11.3% of a diagnostic sample of the shaikhsalman text was mostly Latin script. No filter was applied before training, so English text is part of the training corpus.
  9. Repeated cross-lingual text: The Urdu, Persian and Arabic part of the corpus is 89,449 unique sentences repeated four times. Its share by character count is 16.1%, but its unique content is much smaller.
  10. Possible overlap with baseline training data: The SindhiLM model card names the ambile-official Sindhi Mega Corpus as training data, and that corpus lists Sindhi Wikipedia among its sources. SindhiLM may therefore have seen text from our Wikipedia evaluation set, which would favor that baseline, not ours.

Complete Documentation Index

Document Path Purpose
Root Guide & Navigation Hub README.md Central repository overview, quickstart, and full benchmark summary
Research Manuscript Draft docs/paper.md Draft for future publication; not yet published. Includes methodology and 18 benchmark rows
Technical Explainer Log docs/explainer.md In-depth engineering explainer into empirical failures, metric corrections, and design rationale
Dataset Provenance Card DATA_SOURCES.md Complete documentation of all training corpora and evaluation sets, licensing, and volumes
Evaluation Module Guide evaluation/README.md Technical documentation of metrics algorithms, CLI arguments, and Python API
Training Pipeline Guide training/README.md Exact SentencePiece training recipe, filtering thresholds, and negative findings
Benchmark Result Summaries results/README.md Detailed register breakdowns across FLORES+, Wikipedia, Sentiment, and Sindh Line
Notebook Reproduction Guide notebooks/README.md Instructions for Kaggle/Colab execution of training and benchmarking pipelines
Interactive Demo Guide demo/README.md Local execution and Hugging Face Spaces deployment guide for Gradio UI
Citation Specification CITATION.cff Machine-readable CFF citation metadata
Project Packaging pyproject.toml Modern build-system and packaging metadata

References and Further Reading

  1. Kudo, T. & Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. EMNLP 2018 System Demonstrations. https://aclanthology.org/D18-2012/
  2. Kudo, T. (2018). Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. ACL 2018. https://aclanthology.org/P18-1007/
  3. Conneau, A., et al. (2020). Unsupervised Cross-lingual Representation Learning at Scale (XLM-RoBERTa). ACL 2020. arXiv:1911.02116
  4. NLLB Team et al. (2022). No Language Left Behind: Scaling Human-Centered Machine Translation (FLORES-200). arXiv:2207.04672
  5. Qwen Team (2024). Qwen2.5 Technical Report. arXiv:2412.15115
  6. Meta AI (2024). Llama 3.2 Model Card & Architecture Overview. https://huggingface.co/meta-llama

Note: For the complete list of 21 baseline model citations and references, please refer to docs/paper.md.


Team and Acknowledgments

This research and tokenizer implementation were developed at the Department of Computer Science, Shaikh Ayaz University, Shikarpur, Pakistan, under the supervision of Dr. Attaullah Sahito (academic supervisor, not an author).

Authors and Roles


License and Citation

License

  • Software Code: Licensed under the open-source MIT License.
  • Trained Tokenizer Models: The Hugging Face listing states CC BY-SA 4.0. Source datasets are reported by the project lead to be free to use, mostly under MIT and other open licenses; exact terms and attribution conditions vary by source and are summarized in DATA_SOURCES.md.
  • Corpus Texts: Refer to DATA_SOURCES.md for original source licenses.

Citation

If you use this tokenizer, benchmark code, or findings in your academic research or applications, please cite:

@misc{nawaz2026sindhitokenizer,
  author = {Nawaz, Ali and Fareed, Muqadas and Sadaf, Soonh},
  title  = {A Corpus-Driven Sindhi Subword Tokenizer: Design, Empirical Validation, and Benchmark Comparison Against Fourteen Existing Tokenizers},
  year   = {2026},
  howpublished = {\url{https://github.com/alinawazcode/sindhi-tokenizer}},
  note   = {Department of Computer Science, Shaikh Ayaz University, Shikarpur, Pakistan}
}

Frequently Asked Questions (FAQ)

1. Why choose the 64k model over 32k or 150k? The 64,000 vocabulary size represents the empirical sweet spot between fertility efficiency (1.147 tokens/word) and embedding matrix parameter overhead. Scaling to 100k or 150k provides marginal fertility gains (1.133 and 1.125 respectively) but more than doubles the vocabulary table size, while the 32k model exhibits one minor loss on the standardized FLORES+ benchmark.
2. What does the round-trip length check measure? It applies Unicode NFKC, removes zero-width formatting characters and whitespace, then compares normalized input and decoded-output lengths. This detects shortening but can miss substitutions or reorderings that preserve length; 0.0000% is not proof of exact text identity.
3. Why is there a 15% mix of Urdu, Persian, and Arabic in the training corpus? Sindhi shares the Perso-Arabic alphabet and borrows significant religious, legal, and cultural vocabulary from Arabic and Persian, as well as shared regional terms with Urdu. This anchor (16.1% of characters, from 89,449 unique sentences repeated four times) was added by design to expose the subword vocabulary to shared Perso-Arabic root morphemes.
4. Why should I avoid transformers.LlamaTokenizer for loading this model? Certain versions of Hugging Face Transformers' LlamaTokenizer fail to load custom SentencePiece Unigram models correctly, silently substituting an empty or 3-token fallback vocabulary without throwing an exception. Always use sentencepiece.SentencePieceProcessor directly or via our verified SPMTokenizerAdapter.

Metadata

Release files for sindhi-tokenizer 1.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sindhi-tokenizer 1.1.0
File Size Uploaded Provenance Actions
sindhi_tokenizer-1.1.0.tar.gz 49.5 kB With provenance Details for sindhi_tokenizer-1.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for sindhi-tokenizer 1.1.0
File Interpreter ABI Platform Actions
sindhi_tokenizer-1.1.0-py3-none-any.whl Python 3 none any Details for sindhi_tokenizer-1.1.0-py3-none-any.whl

Total release size: 72.6 kB

Release files / sindhi_tokenizer-1.1.0.tar.gz

Download URL sindhi_tokenizer-1.1.0.tar.gz
Size 49.5 kB
Tags Source
SHA-256 checksum
How to use checksums
4be6e34ac88c27194402ad40accb3980008debb307ab72c3a159d3a937e77551
BLAKE2b-256 checksum
How to use checksums
8774f31fdfdd55fca357c9cf331da59f273373381fe1ef89d63a11835ee81ebb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Release files / sindhi_tokenizer-1.1.0-py3-none-any.whl

Download URL sindhi_tokenizer-1.1.0-py3-none-any.whl
Size 23.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b6163373d67cac3b21033f2f02c93d37922bdc61f5862a6b5ec52504ce00f3ba
BLAKE2b-256 checksum
How to use checksums
817a504e704e8dc74e995009d2d4566b410a31b813f262ef8aba8a3ed2f4ea72
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Security for sindhi-tokenizer 1.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for sindhi_tokenizer-1.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page