polystring

Span-level language detection for mixed-language text.

Most language detectors return one label for a whole string. polystring returns a labelled span for every part of it, with character offsets, a confidence and the script it is written in, so a sentence that switches language mid-way is not flattened into a single wrong guess.

import polystring

for span in polystring.detect("je ne peux pas venir demain, can you call me later?").spans:
    print(f"{span.lang:8} {span.confidence:.2f}  {span.text!r}")

fr       1.00  'je ne peux pas venir demain, '
en       0.94  'can you call me later?'

Install

pip install polystring

Python 3.9 or newer. The model ships with the package as data files, so the first call works offline.

Usage

import polystring

detection = polystring.detect("Здравствуйте, my friend and 你好")

for span in detection.spans:
    print(span.start, span.end, span.lang, span.script, round(span.confidence, 2))

0   14  ru  Cyrl  0.74
14  28  en  Latn  0.66
28  30  zh  Hani  0.55

Reuse a Detector when you are calling it repeatedly; the model is loaded once and shared.

detector = polystring.Detector(abstain_below=0.5)
results = [detector.detect(line) for line in lines]

Detection

Span

Spans always tile the input: they are sorted, non-overlapping, and together cover [0, len(text)). URLs, mentions, hashtags, emoji and bare numerals come back as SPECIAL spans and are never given a language. Where the evidence does not support an answer, the span is UND rather than a guess.

Detector options

Errors

All exceptions derive from polystring.PolystringError.

How it works

Three stages, and which one answers explains most of the accuracy.

Extraction claims the parts of the text that are not language: URLs, mentions, hashtags, emoji, numerals.

Script segmentation splits the rest into runs of a single script. A script used by only one language is answered here, with no model call, because there is nothing to disambiguate. A script shared by several languages becomes a window for the next stage, carrying the language it resolves to on script evidence alone. Perso-Arabic and Cyrillic have marker-codepoint tables, since some of their languages have exclusive letters.

Disambiguation decides the windows that remain. It shortlists the languages plausibly present in the window, scores each token against that shortlist, and decodes the result as a sequence, so a token's neighbours can outweigh its own weak evidence and a genuine one-word switch still survives.

The classifier is a multinomial Naive Bayes model over character n-grams, with two further sources of evidence folded into the same score: whole-word probabilities, and an orthographic penalty for a language that does not write a letter present in the text. A separate term asks whether the input is a language at all, which is what makes keyboard mash come back as UND.

Coverage

394 language and script pairs across 30 scripts. Labels are canonical <ISO 639-3>_<ISO 15924> internally and BCP-47 tags at the API boundary, and the two round-trip:

from polystring import inventory

inventory.normalize("ur-Latn")  # 'urd_Latn'
inventory.to_legacy("urd_Latn")  # 'ur-Latn'
inventory.scripts_for("urd")  # ('Arab', 'Latn')

Because language and script are separate axes, Urdu in Nastaliq and Roman Urdu are different labels with different difficulty, and span.orthography tells them apart without string-matching the tag.

Romanized text

Romanized labels are detected and allocated to their language, covering the informal writing and typing patterns people use in several languages:

polystring.detect("hava cok guzel").spans[0].lang  # 'tr'
polystring.detect("kya tum mere saath chaloge").spans[0].lang  # 'hi-Latn'

Confusion groups

Some labels cannot be reliably told apart, and splitting a dialect continuum finely would otherwise make the detector abstain more, not less. Abstention is decided on the pooled confidence of the winner's group, while the reported confidence stays with the individual label:

from polystring import confusion

sorted(confusion.group_of("hin_Latn"))  # ['hif_Latn', 'hin_Latn', 'urd_Latn']

Accuracy

Measured on held-out text with no training overlap, one sample per language so that a common language cannot carry the average.

Against the other Python detectors, using the same held-out text for each. Every system is scored only on the languages it actually supports, so a missing label counts against coverage rather than against accuracy:

The test set is drawn from the 74 languages both lingua and polystring support. langdetect has labels for 50 of them and langid for 66, and each is scored on its own subset.

"Finds both" splices two languages into one line and asks for both back. langid and langdetect return one label per document by design, so the column does not apply to them.

Development

git clone https://github.com/saadlohani/polystring
cd polystring
pip install -e ".[dev]"
pytest

Linting and typing are enforced in CI:

ruff format --check src/ tests/
ruff check src/ tests/
mypy --strict src/polystring/

The same formatting and lint checks run locally as pre-commit hooks:

pre-commit install

The corpus and the scripts that build the model from it are not in this repository; the trained artifacts in src/polystring/data/ are what the library needs, and they are versioned with it.

License

MIT. See LICENSE.