Skip to main content

hindko-tokenizer

The Hindko tokenizer as a Python package. Install it from PyPI and start tokenizing: no repository clone and no download, because the tokenizer file ships inside the package.

Hindko (Northern hno, Southern hnd) is the Indo-Aryan language of Peshawar, Hazara and neighbouring districts of Pakistan, written in Urdu (Perso-Arabic) script. This is a 32,768-id SentencePiece Unigram tokenizer trained on Hindko newspapers, books and web text.

pip install hindko-tokenizer

Scorecard

Quick start

from hindko_tokenizer import HindkoTokenizer

tokenizer = HindkoTokenizer.from_pretrained()

text = "ایہہ بہت کہٹ لوک جانڑدین کہ ڈرامے دی پیدائش کسراں تے کتھے ہوئی آئی۔"
print(tokenizer.tokenize(text))
print(tokenizer.encode(text))
print(tokenizer.decode(tokenizer.encode(text)))   # gives back the same text

What you can do

tokenizer.tokenize(text)          # token strings; "▁" marks a preceding space
tokenizer.encode(text)            # token ids (no special tokens are added)
tokenizer.decode(ids)             # ids back to text
tokenizer.count_tokens(text)      # how many tokens a text costs

tokenizer.encode_batch(texts)     # many texts at once (parallel)
tokenizer.decode_batch(list_of_ids)

tokenizer.vocab_size              # 32768
tokenizer.token_to_id("\n")       # 64

Normalizing input

The tokenizer itself changes nothing in your text. User-typed or pasted text often has extra spaces, invisible characters or Arabic-keyboard letters (ي ك instead of ی ک), and these cost extra tokens. normalize() puts text into the form the tokenizer was trained on:

from hindko_tokenizer import normalize

clean = normalize(raw_text)
ids = tokenizer.encode(raw_text, normalize=True)               # same as encode(normalize(raw_text))
tokenizer = HindkoTokenizer.from_pretrained(normalize=True)    # or switch it on for every call

normalize() is deterministic and idempotent. It is not reversible, so after it decode() returns the normalized text. Do not use it where whitespace must stay exactly as it was. It is the frozen release 1.0.1 used for all training and evaluation text.

Use with Hugging Face transformers

pip install "hindko-tokenizer[transformers]"
hf_tokenizer = tokenizer.to_transformers()
hf_tokenizer(text)["input_ids"]   # the same ids as tokenizer.encode(text)

Load from a file

tokenizer = HindkoTokenizer.from_pretrained("path/to/folder")   # a folder with tokenizer.json, or the file itself
tokenizer.backend                                               # the underlying tokenizers.Tokenizer

Results

Measured on the held-out test split of the Hindko corpus (details and every source on the model card):

External comparison Fewest tokens of the 131 distinct lossless tokenizers compared
Nearest external tokenizer ProximaAI/urnova-95m: this tokenizer uses 16.7 % fewer tokens
Frontier tokenizers 36.6 % fewer tokens than Gemma 3/4, 40.4 % fewer than GPT-4o, 48.8 % fewer than Llama 4, 50.6 % fewer than Qwen 3.5, 53.7 % fewer than DeepSeek-V3

These gains apply to a model that is trained or adapted with this tokenizer. The tokenizer cannot be plugged into an existing model such as GPT-4o or Llama.

License

CC BY 4.0: free to use, share and adapt, including commercially, with attribution. The license covers the tokenizer and this package, not the texts it was trained on.

Metadata

Release files for hindko-tokenizer 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hindko-tokenizer 1.0.0
File Size Uploaded Provenance Actions
hindko_tokenizer-1.0.0.tar.gz 382.5 kB Without provenance Details for hindko_tokenizer-1.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for hindko-tokenizer 1.0.0
File Interpreter ABI Platform Actions
hindko_tokenizer-1.0.0-py3-none-any.whl Python 3 none any Details for hindko_tokenizer-1.0.0-py3-none-any.whl

Total release size: 768.6 kB

Release files / hindko_tokenizer-1.0.0.tar.gz

Download URL hindko_tokenizer-1.0.0.tar.gz
Size 382.5 kB
Tags Source
SHA-256 checksum
How to use checksums
389b6e5545f719b4fdd31d8c916c1eb54274c12050cdcc12f712ce50b0b06f29
BLAKE2b-256 checksum
How to use checksums
4d2b9da7e46c853227b63470aa3a85f0b3ac6687ee7fd32928dc74c9dbbae3a8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.9

Release files / hindko_tokenizer-1.0.0-py3-none-any.whl

Download URL hindko_tokenizer-1.0.0-py3-none-any.whl
Size 386.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
616adcd592fe815d6c619150f63987b4822fdda846b9f55bc347d8bfe63a090c
BLAKE2b-256 checksum
How to use checksums
1a52c2395e79dd3a86b7c5ea28d9c0f86c1d07ad9a41f6db8db8076e738b3cb3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.9

Security for hindko-tokenizer 1.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page