laghu

A model2vec static embedding model loaded with laghu (Sanskrit for light) has encode, encode_as_sequence and tokenize. Its output matches model2vec’s bit for bit, apart from the empty-input difference documented below.

model2vec reads the embedding table into each process. For potion-multilingual-128M, the table takes 512 MB. Its Unigram tokenizer adds a character trie over 500,353 pieces, about another 610 MB. After encoding, the process uses about 1.3 GB of private memory.

laghu maps the table read-only from model.safetensors. Processes share the mapped pages through the operating system’s page cache. The operating system can reclaim clean pages under memory pressure. For supported Unigram models, laghu replaces the trie with Python Viterbi over a dict of pieces. The normalizer and pre-tokenizer come from tokenizers, built from the same tokenizer.json with the vocabulary removed. The same model then takes about 135 MB of private memory per process.

WordPiece, BPE and word-level models still use tokenizers. laghu maps their embedding tables but does not replace their tokenizers. The memory saving depends on the table’s size relative to the tokenizer.

Install

uv add laghu        # or: pip install laghu

model2vec itself is not a dependency. For the fallback described below, and for same_as_model2vec, install the extra: uv add 'laghu[fallback]'.

Use

from laghu import load

m = load('minishlab/potion-multilingual-128M')      # a folder, or a Hugging Face repo id (the local cache is tried first)
v = m.encode(['The quick brown fox.', 'कर्मण्येवाधिकारस्ते'])   # float32, (2, 256), unit rows: the same array model2vec returns
m.tokenize(['hello world'])
m.encode_as_sequence('hello world').shape          # (n_tokens, 256)

(2, 256)

load returns a LeanModel. If loading fails, it logs the cause and tries model2vec’s StaticModel, which needs laghu[fallback]. If the fallback also fails, load raises the original laghu error. load(..., fallback=False) raises without trying model2vec. Both loaders refuse bfloat16 and float8 tables because numpy cannot hold those dtypes.

same_as_model2vec(path) loads a model both ways and compares them on the built-in corpus, or on texts of your own:

from laghu import same_as_model2vec
same_as_model2vec('minishlab/potion-base-8M', ['my', 'own texts'])

/Users/71293/code/personal/orgs/laghu/.venv/lib/python3.13/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

{'lean': False,
 'n': 2,
 'ids': True,
 'dtype': True,
 'f32': True,
 'f16': True,
 'exact': True,
 'seq': True,
 'median': True,
 'unk': True,
 'tokens': True}

It compares the token ids and the vectors cast to float32 and to float16. It also compares the exact dtype and bits, encode_as_sequence, median_token_length, the unknown token id and the token list.

What matches, and what it costs

Every model below matches model2vec’s ids and vectors, bit for bit, on 4,726 texts. The corpus includes prose in 36 languages and random strings from 30 Unicode blocks. It also tests control characters, special tokens, emoji sequences, combining marks, bidi controls and a 200,000-character string. Each memory measurement uses a fresh process. On macOS the helper reports phys_footprint after encoding and lifetime_max_phys_footprint for peak. On Linux it reports RssAnon and peak resident memory (VmHWM). Speed is encode over 3,226 texts on an M3 Max.

Unigram throughput ranges from about a third of model2vec’s rate to about the same. tokenizers parallelizes batches across cores. Python Viterbi runs on one core. These batch measurements do not establish latency for one query.

The word cache helps when the pre-tokenizer splits words, as in jina-v3, bge-m3 and multilingual-e5. Natural text runs 1.6 to 1.9 times faster on a first pass and 3 to 5 times faster on repeated text. The bench notebook records these measurements and the reasons for rejecting tokie and gigatoken.

Other models use the same tokenizers code in both loaders. The timing samples last about 0.1 seconds and vary by up to half on repeated runs. They do not establish a speed advantage.

What is not supported

• Quantizing or truncating on load (quantize_to, dimensionality, vocabulary_quantization): each would copy the table, which is what laghu exists to avoid. Load with model2vec for those.
• single_word added tokens on a Unigram model fall back to tokenizers for tokenizing (the table is still mapped). Rust decides a word boundary by Unicode’s Alphabetic property, which Python’s str.isalpha does not match.
• use_multiprocessing is accepted and ignored; the result is the same.
• encode([]) returns an empty (0, dim) array. model2vec raises.

Developing

nbdev 3 on hatchling: the notebooks in nbs/ are the source. uv run nbdev-prepare exports, tests and rebuilds this README; uv run nbdev-test --flags slow also runs the bench, which downloads about 5 GB of models.

Licence

MIT.