MillID

Detect the language of any text, offline, in Python. MillID recognises 1,550 languages written in 56 scripts, including hundreds of African, Asian and Indigenous languages that common language detectors do not cover. It runs on a CPU with no PyTorch and no server; the smallest model is 11 MB.

from millid import MillID

model = MillID.from_pretrained("tiny-v1.5")
model.predict("Mo fẹ́ràn láti ka ìwé ní àkókò òmìnira mi.", k=1)
# [Prediction(label='yor_Latn', language='Yoruba', probability=0.973627)]

What is MillID?

MillID (MultiLingual Language IDentifier) is an open language identification model built by Baobab Tech. Give it a piece of text, from a short sentence to a whole document, and it tells you which language the text is written in, with a probability for each answer. Every answer names a language and its script, so Serbian in Cyrillic (srp_Cyrl) and Serbian in Latin (srp_Latn) are told apart.

Developers use it to:

• send each document to the right translation, OCR or speech model;
• sort or filter text by language when building a dataset;
• show an interface in the reader's language, or flag text in an unexpected one;
• count which languages a collection of documents contains.

What this package does

This package runs MillID on your own machine. The first time you load a model it downloads the model's files from the Hugging Face Hub into the Hugging Face cache; after that it works offline. The model runs in ONNX Runtime, on a CPU or an NVIDIA GPU, with NumPy for the text encoding. There is no API to call and no PyTorch to install, and the text you identify never leaves your machine.

It also gives you a command-line tool and, if you want one, an HTTP service. The JavaScript package @baobabtechai/millid gives the same answers in Node.js and the browser.

Install

pip install "millid[cpu]"           # ONNX Runtime for the CPU
pip install "millid[gpu]"           # ONNX Runtime for NVIDIA GPUs (CUDA 12, its libraries from pip)
pip install "millid[cpu,server]"    # and the HTTP service

Python 3.10 or later. ONNX Runtime comes as an extra because its CPU and GPU builds cannot be installed side by side: choose one.

Quick start

from millid import MillID

model = MillID.from_pretrained("tiny-v1.5")     # downloads 11 MB once

model.predict("Je voudrais réserver une table pour deux personnes ce soir.", k=2)
# [Prediction(label='fra_Latn', language='French', probability=0.954421),
#  Prediction(label='frp_Latn', language='Arpitan', probability=0.001443)]

model.predict(["Lave men ou ak savon anvan ou manje.", "Ndimakonda kuwerenga mabuku nthawi yanga yaulere."], k=1)
# [[Prediction(label='hat_Latn', language='Haitian Creole', probability=0.978552)],
#  [Prediction(label='nya_Latn', language='Nyanja', probability=0.940327)]]

predict(text, k) returns the k most likely languages (3 by default), most likely first. Pass a list to identify many texts in one call: they are batched together, which is much faster than one call per text.

Command line

millid predict "Lave men ou ak savon anvan ou manje." -k 2
# hat_Latn 0.979, crs_Latn 0.000	Lave men ou ak savon anvan ou manje.

cat sentences.txt | millid predict -k 1                     # one text per line
millid predict --document < report.txt                      # the whole file as one document, as JSON
millid predict --model large-v1.5 --device cuda "..."       # another model, on the GPU

Choose a model

Model	Download	Top-1 accuracy	Macro F1	Best for
tiny-v1.5 (default)	11 MB	95.9%	0.955	any CPU, laptops, small servers
medium-v1	455 MB	96.2%	0.958	servers
large-v1.5	764 MB	96.7%	0.965	servers with a GPU: the most accurate

Measured on 240,241 test passages in 1,621 languages, from documents the models never saw in training. Macro F1 weighs every language equally, so it shows how well the small languages are served, not just the large ones. The leaderboard compares the three with GlotLID and Meta's fastText identifier, language by language.

from_pretrained also takes a Hugging Face repo id ("baobabtech/millid-large-v1.5") or a local directory holding config.json, labels.json and onnx/model.onnx, and these options:

model = MillID.from_pretrained(
    "large-v1.5",
    device="cuda",       # "cpu" (the default), "cuda", or "auto": the GPU when there is one
    threads=4,           # CPU threads for ONNX Runtime; its default when omitted
    batch_size=64,       # texts per model call
)

Reading the results

Each answer is a Prediction with three fields:

Field	Example	Meaning
label	'yor_Latn'	the language's ISO 639-3 code and its ISO 15924 script code
language	'Yoruba'	the language's English name
probability	0.973627	how likely this answer is, from 0 to 1; the answers for a text add up to at most 1

A few labels end in a script alone, such as und_Tibt: text in that script whose language MillID cannot name more precisely.

Long text and whole documents

predict reads the first 255 bytes of a text, about 250 characters of English or 85 of Hindi. For anything longer, use identify_document: it reads the whole text in windows, gives the document's languages weighted by how much of the text each one covers, and marks where each language runs.

doc = model.identify_document(long_text, k=2)
doc.languages
# [Prediction(label='eng_Latn', language='English', probability=0.576472),
#  Prediction(label='fra_Latn', language='French', probability=0.381211)]
doc.segments
# [Segment(start=0, end=250, label='eng_Latn', language='English', probability=0.952214),
#  Segment(start=251, end=414, label='fra_Latn', language='French', probability=0.96499)]
doc.windows, doc.characters
# (2, 414)

start and end count characters in the text after normalisation (millid.normalize(text)), which collapses whitespace and applies Unicode NFKC.

An HTTP service

millid serve --models "tiny-v1.5=baobabtech/millid-tiny-v1.5@cpu" --port 8050
curl -s localhost:8050/identify -H 'content-type: application/json' -d '{"text": "Lave men ou ak savon anvan ou manje.", "k": 1}'
# {"model": "tiny-v1.5", "results": [{"languages": [{"label": "hat_Latn", "language": "Haitian Creole", "probability": 0.978552}]}]}

POST /identify takes text or texts, k, model, and document: true for a whole document; GET /models describes the loaded models and GET /health reports that the service is up. Requests that arrive together are answered in shared batches, so throughput grows with load. To serve MillID inside an application of your own, build on millid.Registry and millid.MicroBatcher.

Tips

• Give it a sentence or more. A word or a short greeting carries little evidence: "Bonjour tout le monde" alone comes back as nrm_Latn (Norman) rather than French. A full sentence or more gives much more reliable answers.
• Treat a low probability as unsure. When the first answer is below 0.5, look at the next ones too, or ask for more text.
• Close languages share their answers. Swahili and Congo Swahili, or Bosnian and Croatian, differ by a few words; a short passage of either often gets both in its top answers. Ask for k=3 and decide with what you know about your text.
• Pass the text as it is. Accents, tone marks and the script are much of what the model reads: do not strip, transliterate or romanise it first.
• One text alone and the same text in a batch can score a little differently. A batch is lower-cased as one string; call predict with one text where scores must not depend on their neighbours.

API

MillID.from_pretrained(ref="tiny-v1.5", revision=None, token=None, **options)	Load a model: "tiny-v1.5", "medium-v1", "large-v1.5", a Hugging Face repo id, or a directory. token only for a private repo; options device, threads, batch_size, gpu_mem_limit_mb
model.predict(text, k=3)	The k most likely languages of a text, as Predictions. A list of texts gives one list per text
model.identify_document(text, k=3, max_windows=256)	A text of any length, as a DocumentResult: languages, segments, windows, characters
model.probabilities(texts)	A NumPy array, one row per text, a probability for each label in model.labels
millid.MODELS	The released models by short name
millid.normalize, millid.encode_batch	The model's input, to run the ONNX file elsewhere

MillID is millid.LanguageIdentifier under a shorter name. A model is safe to call from several threads at once.

How it works

MillID is a small transformer that reads text as UTF-8 bytes rather than words, so there is no vocabulary to outgrow: any script, even one it has rarely seen, is input it can read. Each byte is described by the byte itself, the Unicode script of its character and the character sequences ending there; the network turns a passage into one vector and scores it against a vector for every label, built from the label itself, its language (shared across scripts) and its script. The full design is in the architecture document, and each model's card on the Hub has its evaluation.

Links

• Models: tiny v1.5, medium v1, large v1.5
• Leaderboard: baobabtech/millid
• Dataset: baobabtech/millid-dataset
• JavaScript package: @baobabtechai/millid on npm
• Source, training code and issues: github.com/baobab-tech/millid

Licence

Apache-2.0, for the code and the model weights. Built by Baobab Tech.