Skip to main content

hindko-embeddings

Hindko text embeddings as a Python package. One install gives you semantic search, similarity, clustering and classification for Hindko text.

Hindko (Northern hno, Southern hnd) is the Indo-Aryan language of Peshawar, Hazara and Kohat in Khyber Pakhtunkhwa, Pakistan, written in Urdu (Perso-Arabic) script. The model turns a Hindko headline, sentence, verse or passage into a 1,024-dimensional vector, so that texts about the same thing end up close together.

pip install hindko-embeddings

Scorecard

Quick start

from hindko_embeddings import HindkoEmbedder

embedder = HindkoEmbedder.from_pretrained()   # first run downloads the model (1.1 GB), then it is cached

passages = [
    "ایہہ بہت کہٹ لوک جانڑدین کہ ڈرامے دی پیدائش کسراں تے کتھے ہوئی آئی۔",
    "اج اساں اگر آپڑیں بزرگاں دی انہاں گلاں تے غور نہ کیتا تے پچھتاوے دے سوا کج ہتھ نہ آسی۔",
    "اسی طراں خیبر پختونخوا اچ بولی جانڑیں ولی دیگر زباناں دے ادب و ثقافت دی بی نمائندگی کیتی گئی اَئی۔",
]

for hit in embedder.search("ڈرامے دی پیدائش", passages, top_k=3):
    print(round(hit["score"], 3), hit["text"])

The SDK applies the two conventions the model was trained with, so you don't have to:

  • it puts text in canonical form with normalize() from hindko-tokenizer, which is installed with this package;
  • it adds the query: or passage: prefix the model expects.

What you can do

q = embedder.encode_queries(["ڈرامے دی پیدائش"])   # search queries, headlines, titles
p = embedder.encode_passages(passages)             # documents to be found
scores = embedder.similarity(q, p)                 # cosine similarity matrix

emb = embedder.encode(texts)                       # similarity, clustering, classification ("query" side)
vector = embedder.encode("ڈرامے دی پیدائش")         # one string gives one vector

hits = embedder.search(query, passages, top_k=5)                         # ranked [{index, score, text}]
hits = embedder.search(query, passages, passage_embeddings=p)            # reuse embeddings across queries

embedder.prepare(texts, kind="passage")   # see exactly what the model receives
embedder.model                            # the underlying SentenceTransformer

from_pretrained() takes device="cuda", a pinned revision= and a cache_folder=. On a CPU the float16 weights are cast to float32 automatically.

Results

On the held-out test split of HindkoEmbedBench v1.0 (five tasks: headline retrieval, book-title retrieval, passage continuation, verse/prose classification and book clustering):

Model Test score
hindko-embeddings 0.7391
Character n-gram TF-IDF 0.6864
bge-m3 (best public embedding model tested) 0.6332
multilingual-e5-large (its base model, not fine-tuned) 0.5977

It has the highest overall test score of the 12 models scored on test, and it is ahead of TF-IDF on four of the five tasks (behind it on book-title retrieval). Every number, chart and the full protocol are on the model card.

Requirements

Python 3.10 or later, sentence-transformers 6.0 or later and transformers 5.x. Older versions may not load the saved model configuration. Input longer than 512 tokens is truncated.

License

CC BY-NC 4.0 (non-commercial): open for research, teaching and other non-commercial use. Commercial use needs written permission from the author. The base model, intfloat/multilingual-e5-large, is MIT-licensed; its notice is kept in LICENSE.

Metadata

Release files for hindko-embeddings 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hindko-embeddings 1.0.0
File Size Uploaded Provenance Actions
hindko_embeddings-1.0.0.tar.gz 12.2 kB Without provenance Details for hindko_embeddings-1.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for hindko-embeddings 1.0.0
File Interpreter ABI Platform Actions
hindko_embeddings-1.0.0-py3-none-any.whl Python 3 none any Details for hindko_embeddings-1.0.0-py3-none-any.whl

Total release size: 25.0 kB

Release files / hindko_embeddings-1.0.0.tar.gz

Download URL hindko_embeddings-1.0.0.tar.gz
Size 12.2 kB
Tags Source
SHA-256 checksum
How to use checksums
922142ea8927a8b18353217bdfa7bba2f03189a1ac6a6d3c5f833cadd110de51
BLAKE2b-256 checksum
How to use checksums
2633d6b978568f6591fadf0444efb8b6ea685800087617db7a067cfbc87a0592
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.9

Release files / hindko_embeddings-1.0.0-py3-none-any.whl

Download URL hindko_embeddings-1.0.0-py3-none-any.whl
Size 12.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c1a147164d1d701765ff91809a4fa6c8522719ecb902c12dab3b19d21ac96a0b
BLAKE2b-256 checksum
How to use checksums
ba02d97ce0d6d451d1cb5f9d702bec46143d833c64a8937cdb9b9a4383e53357
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.9

Security for hindko-embeddings 1.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page