noentenc 0.2.0

Language detection and machine translation for Python, on CPU, without torch.

from noentenc import Language
from noentenc.language_detection import LanguageDetector
from noentenc.translation import Translator

LanguageDetector().detect("Bon dia! Com estàs?")
# 'cat'

Translator().translate("The weather is nice today.", Language.SPANISH, Language.ENGLISH)
# 'El tiempo es bueno hoy.'

• Fast. Detects a language in as little as 2.5 µs and translates a sentence in about 52 ms, on a laptop CPU. See benchmarks.
• Faster than the originals. Up to 470× faster than langid.py, 2× faster than the C++ fasttext package and 5.8× faster than transformers + PyTorch, with the same accuracy. See the comparison.
• Small. The default detection model is 0.9 MB. noentenc has no torch, transformers or GPU dependency, only numpy, onnxruntime, tokenizers, huggingface-hub and tqdm.
• One API, many models. 12 detection models and 4 translation model families sit behind the same two classes. Swapping one is a one-line change, and every detector returns the same ISO 639-3 labels.
• Built for datasets. You can pass a single string, a list or a pandas or polars column.

New to noentenc? The quickstart covers detection, translation and choosing a model on one page.

Install

noentenc uses uv to manage dependencies.

Requires Python 3.14 or newer. Install with pip or uv:

pip install noentenc
pip install 'noentenc[all]'

uv add noentenc                  # fastText, ONNX and langid detection + every translation model
uv add 'noentenc[lingua,cld3]'   # extra detection backends, see the table below
uv add 'noentenc[all]'

Weights download on first use to ~/.cache/noentenc (detection) and the Hugging Face cache (translation). Nothing is bundled in the wheel.

Detect a language

from noentenc.language_detection import LanguageDetector

detector = LanguageDetector()  # fastText lid.176: 176 languages, 0.9 MB

detector.detect("Bon dia! Com estàs?")
# 'cat'
detector.detect("Bon dia! Com estàs?", with_score=True, top_k=2)
# {'cat': 0.88, 'por': 0.11}
detector.detect_batch(["Hello there", "Hola, ¿qué tal?", "你好", ""])
# ['eng', 'spa', 'zho', 'und']

# Add a "lang" column to a pandas or polars DataFrame.
detector.detect_dataset(df, "text", "lang")

Translate

from noentenc import Language
from noentenc.translation import Translator

translator = Translator()

translator.translate_batch(
    ["Where is the station?", "I love this city."], Language.SPANISH, Language.ENGLISH
)
# ['¿Dónde está la estación?', 'Me encanta esta ciudad.']

# The source language is optional.
translator.translate("Bon dia a tothom!", Language.ENGLISH)
# 'Good day to everyone!'

translator.translate_dataset(df, "review", "review_en", Language.ENGLISH)

Translator() picks the lightest model for each pair. It uses a dedicated Opus-MT model when one exists for the direction (66 directions, about 75M parameters each). For any other pair it uses SMaLL-100, which covers 100 languages. Languages are always Language enum members, so a typo fails at the call site, and a model that can't handle a pair raises UnsupportedLanguageError.

Trade speed for quality

LanguageDetector and Translator take a profile in place of a model: "speed" (the default), "balance" or "quality".

from noentenc import Profile
from noentenc.language_detection import LanguageDetector
from noentenc.translation import Translator

LanguageDetector("quality")  # fastText GlotLID
Translator(Profile.BALANCE)  # Opus-MT where it exists, otherwise NLLB-200 at int8

On FLORES-200, balance raises detection accuracy from 50% to 96% over the 176 languages tested, and NLLB-200 adds up to 19 chrF++ on low-resource pairs such as English to Tamil. See the profiles benchmark for the numbers behind each choice. The models behind a profile may change between releases, so pass a model explicitly when you need reproducible output.

Examples

Models

Language detection

from noentenc.language_detection import FastTextModel, LanguageDetector, LinguaModel

LanguageDetector(FastTextModel("glotlid"))  # 2102 languages
LanguageDetector(LinguaModel(languages=["cat", "spa", "eng"]))  # only these candidates

• Labels are ISO 639-3 codes (eng, cat, zho). Empty and whitespace-only texts return und.
  • normalize_labels=False returns each model's native codes instead.
  • collapse_macrolanguages=True folds individual languages into their macrolanguage (arb → ara, cmn → zho), so results from different models line up.
• FastTextModel is our own numpy implementation of fastText inference. It needs neither the fasttext package nor onnxruntime, and it matches fasttext's output to within 1e-6. Large .bin models are memory-mapped, so they open instantly.
• LangidModel is our own numpy implementation of langid.py. It reads the weights from the langid 1.1.6 source release on PyPI, without installing or importing the langid package, and matches its probabilities to within 1e-9.

Translation

Every translation model takes precision= (fp32, int8, q4, where the export has it), num_threads= and a Hugging Face repo id or local directory as model=.

Weights and licences

Weights are pinned to a revision and, where we download them ourselves, checked against a sha256. Set NOENTENC_CACHE to change the detection cache location. With only_local_files=True, a model that isn't cached raises instead of downloading, which is what you want on an air-gapped server.

The weights' licences apply to your use of the weights. They don't affect this package's licence.

Bring your own model

Load your own fastText, ONNX classifier or seq2seq weights into an existing backend, or wrap any detector or translator by subclassing a base class with a few methods. See docs/custom-models.md.

from noentenc.language_detection import FastTextModel, LanguageDetector

LanguageDetector(FastTextModel("models/my-domain-lid.ftz"))

Development

make install             # uv sync with every extra + dev tools
make lint
make unit-tests
make integration-tests   # tests/integrations; downloads real weights

scripts/benchmark_lid.py and scripts/benchmark_translation.py reproduce the speed numbers, and scripts/benchmark_vs_reference.py the comparison with the original implementations. scripts/make_fasttext_fixtures.py regenerates the fastText parity fixtures with the reference fasttext package. That package needs Python 3.12, and the script's docstring has the command. scripts/make_langid_fixtures.py does the same for langid with the reference langid package.