lingua-language-detector 1.3.5

An accurate natural language detection library, suitable for short text and mixed-language text

1. What does this library do?

Its task is simple: It tells you which language some text is written in. This is very useful as a preprocessing step for linguistic data in natural language processing applications such as text classification and spell checking. Other use cases, for instance, might include routing e-mails to the right geographically located customer service department, based on the e-mails' languages.

2. Why does this library exist?

Language detection is often done as part of large machine learning frameworks or natural language processing applications. In cases where you don't need the full-fledged functionality of those systems or don't want to learn the ropes of those, a small flexible library comes in handy.

Python is widely used in natural language processing, so there are a couple of comprehensive open source libraries for this task, such as Google's CLD 2 and CLD 3, Langid, fastText, fastspell, Simplemma and Langdetect. Unfortunately, most of them have two major drawbacks:

1. Detection only works with quite lengthy text fragments. For very short text snippets such as Twitter messages, they do not provide adequate results.
2. The more languages take part in the decision process, the less accurate are the detection results.

Lingua aims at eliminating these problems. She nearly does not need any configuration and yields pretty accurate results on both long and short text, even on single words and phrases. She draws on both rule-based and statistical methods but does not use any dictionaries of words. She does not need a connection to any external API or service either. Once the library has been downloaded, it can be used completely offline.

3. A short history of this library

This library started as a pure Python implementation. Python's quick prototyping capabilities made an important contribution to its improvements. Unfortunately, there was always a tradeoff between performance and memory consumption. At first, Lingua's language models were stored in dictionaries during runtime. This led to quick performance at the cost of large memory consumption (more than 3 GB). Because of that, the language models were then stored in NumPy arrays instead of dictionaries. Memory consumption reduced to approximately 800 MB but CPU performance dropped significantly. Both approaches were not satisfying.

Starting from version 2.0.0, the pure Python implementation was replaced with compiled Python bindings to the native Rust implementation of Lingua. This decision has led to both quick performance and a small memory footprint of less than 1 GB. The pure Python implementation is still available in a separate branch in this repository and will be kept up-to-date in subsequent 1.* releases. Both 1.* and 2.* versions will remain available on the Python package index (PyPI).

4. Which languages are supported?

Compared to other language detection libraries, Lingua's focus is on quality over quantity, that is, getting detection right for a small set of languages first before adding new ones. Currently, the following 75 languages are supported:

• A
  • Afrikaans
  • Albanian
  • Arabic
  • Armenian
  • Azerbaijani
• B
  • Basque
  • Belarusian
  • Bengali
  • Norwegian Bokmal
  • Bosnian
  • Bulgarian
• C
  • Catalan
  • Chinese
  • Croatian
  • Czech
• D
  • Danish
  • Dutch
• E
  • English
  • Esperanto
  • Estonian
• F
  • Finnish
  • French
• G
  • Ganda
  • Georgian
  • German
  • Greek
  • Gujarati
• H
  • Hebrew
  • Hindi
  • Hungarian
• I
  • Icelandic
  • Indonesian
  • Irish
  • Italian
• J
  • Japanese
• K
  • Kazakh
  • Korean
• L
  • Latin
  • Latvian
  • Lithuanian
• M
  • Macedonian
  • Malay
  • Maori
  • Marathi
  • Mongolian
• N
  • Norwegian Nynorsk
• P
  • Persian
  • Polish
  • Portuguese
  • Punjabi
• R
  • Romanian
  • Russian
• S
  • Serbian
  • Shona
  • Slovak
  • Slovene
  • Somali
  • Sotho
  • Spanish
  • Swahili
  • Swedish
• T
  • Tagalog
  • Tamil
  • Telugu
  • Thai
  • Tsonga
  • Tswana
  • Turkish
• U
  • Ukrainian
  • Urdu
• V
  • Vietnamese
• W
  • Welsh
• X
  • Xhosa
• Y
  • Yoruba
• Z
  • Zulu

5. How accurate is it?

Lingua is able to report accuracy statistics for some bundled test data available for each supported language. The test data for each language is split into three parts:

1. a list of single words with a minimum length of 5 characters
2. a list of word pairs with a minimum length of 10 characters
3. a list of complete grammatical sentences of various lengths

Both the language models and the test data have been created from separate documents of the Wortschatz corpora offered by Leipzig University, Germany. Data crawled from various news websites have been used for training, each corpus comprising one million sentences. For testing, corpora made of arbitrarily chosen websites have been used, each comprising ten thousand sentences. From each test corpus, a random unsorted subset of 1000 single words, 1000 word pairs and 1000 sentences has been extracted, respectively.

Given the generated test data, I have compared the detection results of Lingua, fastText, langdetect, langid, CLD 2 and CLD 3 running over the data of Lingua's supported 75 languages. Languages that are not supported by the other detectors are simply ignored for them during the detection process.

Each of the following sections contains two plots. The bar plot shows the detailed accuracy results for each supported language. The box plot illustrates the distributions of the accuracy values for each classifier. The boxes themselves represent the areas which the middle 50 % of data lie within. Within the colored boxes, the horizontal lines mark the median of the distributions.

5.1 Single word detection

Bar plot

5.2 Word pair detection

Bar plot

5.3 Sentence detection

Bar plot

5.4 Average detection

Bar plot

5.5 Mean, median and standard deviation

The table below shows detailed statistics for each language and classifier including mean, median and standard deviation.

Open table

Language	Average	Single Words	Word Pairs	Sentences
	Lingua (high accuracy mode)	Lingua (low accuracy mode)	Langdetect	FastText	FastSpell (conservative mode)	FastSpell (aggressive mode)	Langid	CLD3	CLD2	Simplemma	Lingua (high accuracy mode)	Lingua (low accuracy mode)	Langdetect	FastText	FastSpell (conservative mode)	FastSpell (aggressive mode)	Langid	CLD3	CLD2	Simplemma	Lingua (high accuracy mode)	Lingua (low accuracy mode)	Langdetect	FastText	FastSpell (conservative mode)	FastSpell (aggressive mode)	Langid	CLD3	CLD2	Simplemma	Lingua (high accuracy mode)	Lingua (low accuracy mode)	Langdetect	FastText	FastSpell (conservative mode)	FastSpell (aggressive mode)	Langid	CLD3	CLD2	Simplemma
Afrikaans	79	64	67	36	70	73	30	55	55	-	58	38	38	11	49	50	1	22	13	-	81	62	65	23	67	74	10	46	56	-	97	93	98	74	94	95	80	98	96	-
Albanian	88	80	79	66	66	66	65	55	65	20	69	54	52	35	35	35	33	18	18	21	95	86	84	66	66	66	63	48	77	17	100	99	99	98	98	98	98	98	99	23
Arabic	98	94	97	96	96	96	91	90	67	-	96	88	93	89	89	89	84	79	19	-	99	96	98	98	98	98	90	92	82	-	100	99	100	100	100	100	98	100	99	-
Armenian	100	100	-	100	100	100	94	99	100	22	100	100	-	100	100	100	83	100	100	36	100	100	-	100	100	100	99	100	100	14	100	100	-	100	100	100	100	97	100	14
Azerbaijani	90	82	-	78	69	85	68	81	72	-	77	71	-	57	43	67	36	62	34	-	92	78	-	80	69	90	69	82	82	-	99	96	-	98	94	100	98	99	99	-
Basque	84	75	-	71	71	71	52	62	61	-	71	56	-	44	44	44	18	33	23	-	87	76	-	70	70	70	52	62	69	-	93	92	-	100	100	100	86	92	91	-
Belarusian	97	92	-	85	92	95	85	84	76	-	92	80	-	69	81	87	69	67	42	-	99	95	-	88	94	98	87	86	87	-	100	100	-	98	99	100	99	100	99	-
Bengali	100	100	100	98	98	98	92	99	63	-	100	100	100	94	94	94	92	98	19	-	100	100	100	99	99	99	88	99	69	-	100	100	100	100	100	100	97	99	99	-
Bokmal	58	50	-	-	69	75	13	-	-	50	39	27	-	-	53	55	3	-	-	15	59	47	-	-	70	77	12	-	-	45	77	75	-	-	85	91	23	-	-	90
Bosnian	35	29	-	9	54	65	5	33	19	-	29	23	-	9	54	54	2	19	4	-	35	29	-	10	64	76	4	28	15	-	41	36	-	8	44	64	8	52	36	-
Bulgarian	87	78	72	78	89	92	67	70	66	68	70	56	50	56	80	83	46	45	32	44	91	81	68	81	88	95	62	66	72	67	99	96	97	99	98	99	93	98	93	91
Catalan	70	58	55	57	63	66	38	48	38	59	51	33	26	33	42	44	5	19	4	32	74	60	52	57	63	67	29	42	30	62	87	82	87	83	85	88	81	84	79	81
Chinese	100	100	64	71	71	71	96	92	33	-	100	100	39	46	46	46	90	92	-	-	100	100	55	68	68	68	97	83	2	-	100	100	97	100	100	100	100	100	98	-
Croatian	73	60	73	47	72	81	48	42	51	-	53	36	50	28	62	64	16	26	34	-	74	57	71	42	79	87	38	42	47	-	90	86	98	72	76	93	90	58	73	-
Czech	80	71	71	76	76	80	66	64	74	50	65	54	52	58	61	64	44	39	50	31	84	71	72	79	78	83	69	65	80	44	91	87	88	92	88	92	86	88	91	76
Danish	81	70	71	62	76	78	60	58	59	50	61	45	50	35	56	58	33	26	27	20	84	70	68	57	75	78	61	54	56	47	98	95	94	95	98	99	86	95	94	83
Dutch	77	64	58	78	71	78	64	58	47	58	55	36	27	55	46	55	34	29	11	32	81	61	49	81	70	81	61	47	42	50	96	94	97	100	97	99	98	97	90	92
English	81	63	60	96	96	96	85	54	56	65	55	29	23	90	90	90	84	22	12	27	89	62	59	98	98	98	71	44	55	69	99	97	99	100	100	100	99	97	100	98
Esperanto	84	66	-	76	76	76	44	57	50	-	67	44	-	51	51	51	5	22	7	-	85	61	-	79	79	79	30	51	46	-	98	93	-	100	100	100	96	98	98	-
Estonian	92	83	83	73	73	73	67	70	65	71	80	62	62	50	50	50	37	41	24	44	96	88	87	73	73	73	67	69	73	70	100	99	100	96	97	97	98	99	99	97
Finnish	96	90	93	92	93	93	83	80	77	76	90	77	84	82	82	82	62	58	44	47	98	95	95	96	96	96	88	84	89	81	100	100	100	100	100	100	100	99	98	100
French	89	77	75	83	83	83	71	55	46	65	74	51	48	62	62	62	42	22	12	34	94	82	77	86	86	86	74	49	48	68	99	98	99	99	99	99	98	94	80	94
Ganda	91	84	-	-	-	-	-	-	61	-	79	65	-	-	-	-	-	-	23	-	95	87	-	-	-	-	-	-	62	-	100	100	-	-	-	-	-	-	99	-
Georgian	100	100	-	99	99	99	99	98	100	4	100	100	-	97	97	97	97	99	100	11	100	100	-	99	99	99	100	100	100	2	100	100	-	100	100	100	100	96	100	0
German	89	80	73	89	89	89	81	66	64	72	74	57	49	76	76	76	61	40	27	38	94	84	69	93	93	93	81	62	66	78	100	99	100	100	100	100	100	98	98	99
Greek	100	100	100	99	99	99	100	100	100	75	100	100	100	98	98	98	100	100	100	74	100	100	100	100	100	100	100	100	100	60	100	100	100	100	100	100	100	100	100	92
Gujarati	100	100	100	100	100	100	100	100	100	-	100	100	100	99	99	99	100	99	100	-	100	100	100	100	100	100	100	100	100	-	100	100	100	100	100	100	100	100	100	-
Hebrew	100	100	100	100	100	100	100	-	-	-	100	100	100	99	99	99	100	-	-	-	100	100	100	100	100	100	100	-	-	-	100	100	100	100	100	100	100	-	-	-
Hindi	73	33	67	87	72	88	60	58	77	5	61	11	44	74	53	77	41	34	56	2	64	20	59	88	65	89	47	45	76	4	94	67	99	99	96	99	92	95	99	11
Hungarian	95	90	88	92	92	92	83	76	75	72	87	77	74	80	80	80	64	53	41	58	98	93	91	96	96	96	86	76	85	62	100	100	100	100	100	100	100	99	100	95
Icelandic	93	87	-	65	70	71	66	71	66	64	83	72	-	39	49	50	33	42	26	43	97	92	-	57	64	65	66	70	73	59	100	99	-	98	99	99	99	99	99	90
Indonesian	61	47	80	69	68	77	51	46	62	26	39	25	56	43	52	56	16	26	36	20	61	46	85	68	73	82	54	45	63	26	83	71	100	95	78	93	82	66	88	32
Irish	91	85	-	60	66	69	63	67	66	77	82	69	-	35	41	47	28	42	29	66	94	90	-	57	66	68	64	66	78	76	96	95	-	89	93	93	97	94	92	90
Italian	87	71	77	89	89	89	66	62	44	58	69	42	50	74	74	74	28	31	7	24	92	74	81	92	92	92	70	57	32	57	100	98	99	100	100	100	100	98	93	94
Japanese	100	100	100	87	87	87	86	98	33	-	100	100	99	72	72	72	61	97	-	-	100	100	100	89	89	89	96	96	-	-	100	100	100	100	100	100	100	100	100	-
Kazakh	96	94	-	88	76	91	80	82	77	-	89	88	-	72	52	79	67	62	43	-	98	94	-	90	80	94	78	83	88	-	100	100	-	100	96	100	96	99	99	-
Korean	100	100	100	99	99	99	100	99	100	-	100	100	100	98	98	98	100	100	100	-	100	100	100	100	100	100	100	100	100	-	100	100	100	100	100	100	100	98	100	-
Latin	87	73	-	50	50	50	21	62	46	63	72	49	-	24	24	24	-	44	9	33	93	76	-	41	41	41	2	58	42	63	97	94	-	85	86	86	61	83	88	93
Latvian	93	87	89	82	82	84	83	75	72	45	85	75	76	65	66	69	64	51	33	36	97	90	91	83	84	86	86	77	84	33	99	97	99	97	97	98	98	98	98	65
Lithuanian	95	87	88	81	81	81	80	72	70	66	86	76	72	61	61	61	58	42	30	50	98	89	91	83	83	83	85	75	82	62	100	98	100	99	99	99	99	99	99	88
Macedonian	84	72	87	74	86	93	51	60	60	13	66	52	71	51	77	83	15	30	27	12	86	70	89	72	83	96	44	54	70	11	99	95	100	100	97	99	94	97	84	15
Malay	31	31	-	15	39	52	11	22	18	13	26	22	-	14	36	38	2	11	9	3	38	36	-	19	52	64	9	22	22	10	28	35	-	12	29	54	22	34	23	26
Maori	92	83	-	-	-	-	-	52	61	-	84	64	-	-	-	-	-	22	12	-	92	88	-	-	-	-	-	43	72	-	99	98	-	-	-	-	-	91	98	-
Marathi	85	39	88	80	8	75	80	84	83	-	74	16	76	61	9	61	70	69	65	-	85	30	89	81	15	69	79	84	86	-	96	72	98	99	1	95	91	98	99	-
Mongolian	97	95	-	81	85	89	86	83	78	-	92	88	-	59	66	72	68	63	43	-	99	98	-	86	91	94	90	87	92	-	99	99	-	98	99	100	99	99	100	-
Nynorsk	66	52	-	29	63	70	32	-	54	24	41	25	-	8	42	43	5	-	18	6	66	49	-	18	58	70	16	-	50	22	91	81	-	61	87	96	75	-	93	45
Persian	90	80	81	90	79	92	92	76	61	12	78	62	64	79	57	84	83	57	13	12	94	80	80	92	81	94	94	70	72	5	100	98	100	100	98	99	100	99	99	18
Polish	95	90	89	92	92	92	89	77	75	86	85	77	74	80	80	80	73	51	38	72	98	93	93	97	97	97	93	80	87	87	100	99	100	100	100	100	100	99	99	99
Portuguese	81	69	60	73	81	84	54	53	54	61	59	42	29	47	66	67	19	21	20	26	85	69	54	71	81	85	44	40	48	60	99	95	98	99	96	99	98	97	94	97
Punjabi	100	100	100	100	100	100	100	100	100	-	100	100	100	99	99	99	100	99	100	-	100	100	100	100	100	100	100	100	100	-	100	100	100	100	100	100	100	100	100	-
Romanian	87	72	77	64	64	64	61	53	54	57	69	49	56	38	38	38	31	24	11	34	92	74	78	60	60	60	60	48	53	51	99	94	97	95	95	95	92	88	96	86
Russian	90	78	84	94	94	97	75	71	60	66	76	59	70	86	88	92	60	48	26	54	95	84	87	98	97	99	75	72	68	62	98	92	95	100	98	99	91	93	87	83
Serbian	88	78	-	76	53	76	64	78	69	-	74	62	-	54	47	54	39	63	29	-	90	80	-	76	58	76	63	75	78	-	99	92	-	98	52	98	89	95	99	-
Shona	91	81	-	-	-	-	-	76	65	-	78	56	-	-	-	-	-	51	24	-	96	86	-	-	-	-	-	79	71	-	100	100	-	-	-	-	-	99	99	-
Slovak	84	75	75	65	80	83	68	63	71	68	64	49	51	41	63	64	40	32	38	45	90	78	75	62	81	86	66	61	76	66	99	97	98	91	97	98	97	96	99	93
Slovene	82	67	73	59	75	77	63	63	48	72	61	39	48	32	56	57	33	29	8	48	87	68	72	54	74	78	61	60	42	72	99	93	98	90	96	97	95	99	92	96
Somali	92	85	90	24	51	52	-	69	70	-	82	64	75	4	18	20	-	38	27	-	96	90	94	15	46	48	-	70	83	-	100	100	100	52	89	89	-	100	99	-
Sotho	86	72	-	-	-	-	-	49	54	-	67	43	-	-	-	-	-	15	13	-	90	75	-	-	-	-	-	33	54	-	100	97	-	-	-	-	-	98	95	-
Spanish	70	56	56	74	64	73	65	48	43	50	44	26	25	51	48	52	37	16	12	16	69	49	46	72	60	74	59	32	34	41	97	94	97	100	85	94	98	96	85	92
Swahili	81	70	73	41	41	41	42	57	57	46	60	43	46	7	7	7	3	25	16	26	84	68	73	24	24	24	24	49	59	41	98	97	99	92	92	92	98	98	97	72
Swedish	84	72	68	76	79	81	65	61	53	59	64	46	41	51	57	59	35	30	14	29	88	76	67	78	82	85	63	56	52	62	99	94	96	98	98	99	96	96	93	87
Tagalog	78	66	76	45	46	46	42	-	50	12	52	36	51	11	11	11	2	-	9	9	83	67	78	28	28	28	26	-	44	11	98	96	99	98	98	98	98	-	95	15
Tamil	100	100	100	100	100	100	100	100	100	-	100	100	100	100	100	100	100	100	100	-	100	100	100	100	100	100	100	100	100	-	100	100	100	100	100	100	100	99	100	-
Telugu	100	100	100	100	100	100	100	99	100	-	100	100	100	100	100	100	100	99	100	-	100	100	100	100	100	100	100	100	100	-	100	100	100	100	100	100	100	99	100	-
Thai	100	100	100	100	100	100	100	99	100	-	100	100	100	100	100	100	100	100	100	-	100	100	100	100	100	100	100	100	100	-	100	100	100	100	100	100	100	98	100	-
Tsonga	84	72	-	-	-	-	-	-	61	-	66	46	-	-	-	-	-	-	19	-	89	73	-	-	-	-	-	-	68	-	98	97	-	-	-	-	-	-	97	-
Tswana	84	71	-	-	-	-	-	-	56	-	65	44	-	-	-	-	-	-	17	-	88	73	-	-	-	-	-	-	57	-	99	96	-	-	-	-	-	-	94	-
Turkish	94	87	82	86	86	86	67	69	66	76	84	71	62	70	70	70	50	41	30	55	98	91	84	88	88	88	67	70	71	78	100	99	100	100	100	100	84	97	97	96
Ukrainian	94	88	83	91	95	98	76	81	77	78	85	75	66	78	90	94	54	62	46	62	98	92	85	94	95	98	77	83	88	75	99	96	98	100	100	100	96	98	99	97
Urdu	90	79	83	63	75	80	58	61	61	-	80	65	67	40	59	68	30	39	8	-	94	78	84	50	68	74	46	53	75	-	96	94	97	99	99	99	99	92	99	-
Vietnamese	91	87	93	89	89	89	86	66	63	-	79	75	81	71	71	71	65	26	-	-	94	87	97	97	97	97	93	74	90	-	99	98	100	100	100	100	100	99	100	-
Welsh	91	82	85	64	69	72	49	69	72	69	78	61	69	35	41	46	11	43	34	58	96	87	88	61	71	74	39	66	85	60	99	99	99	96	96	97	95	98	98	90
Xhosa	82	69	-	-	-	-	53	66	71	-	64	45	-	-	-	-	13	40	45	-	85	67	-	-	-	-	49	65	71	-	98	94	-	-	-	-	96	92	97	-
Yoruba	74	62	-	8	8	8	-	15	37	-	50	33	-	1	1	1	-	5	1	-	77	61	-	1	1	1	-	11	22	-	96	92	-	21	22	22	-	28	88	-
Zulu	81	70	-	-	-	-	6	63	54	-	62	45	-	-	-	-	0	35	18	-	83	72	-	-	-	-	6	63	51	-	97	94	-	-	-	-	11	92	93	-
Mean	86	77	82	74	77	81	68	69	65	52	74	61	65	58	62	66	48	48	34	34	89	78	82	74	77	82	65	67	68	50	96	93	98	92	91	96	90	93	94	73
Median	89.0	79.0	82.5	78.0	79.0	83.0	67.0	68.0	63.0	59.0	74.0	57.0	63.0	57.5	61.0	67.0	41.5	41.0	26.5	33.0	94.0	81.0	84.5	81.0	81.0	86.0	67.0	66.0	71.5	60.0	99.0	97.0	99.0	99.0	98.0	99.0	98.0	98.0	98.0	90.0
Standard Deviation	13.12	17.32	13.4	23.07	19.9	17.0	24.61	19.04	18.57	23.46	18.48	25.0	23.59	28.52	25.31	24.22	32.33	27.86	28.74	18.94	13.14	18.95	15.68	26.45	21.67	19.67	28.5	21.83	22.7	24.48	11.05	11.9	2.69	19.46	19.1	11.78	20.21	13.95	12.25	31.91

6. How fast is it?

The accuracy reporter script measures the time each language detector needs to classify 3000 input texts for each of the supported 75 languages. The results below have been produced on an iMac 3.6 Ghz 8-Core Intel Core i9 with 40 GB RAM.

CLD 2 and 3, FastText and FastSpell have at least partially been implemented in C or C++, that is why they are the most performant ones. The others have been implemented in pure Python including Lingua.

Detector	Time
CLD 2	8.65 sec
FastText	10.50 sec
CLD 3	16.77 sec
FastSpell (aggressive mode)	51.92 sec
FastSpell (conservative mode)	52.32 sec
Simplemma	2 min 36.44 sec
Langid	3 min 50.40 sec
Lingua (low accuracy mode)	4 min 05.53 sec
Lingua (high accuracy mode)	8 min 09.52 sec
Langdetect	10 min 43.96 sec

7. Why is it better than other libraries?

Every language detector uses a probabilistic n-gram model trained on the character distribution in some training corpus. Most libraries only use n-grams of size 3 (trigrams) which is satisfactory for detecting the language of longer text fragments consisting of multiple sentences. For short phrases or single words, however, trigrams are not enough. The shorter the input text is, the less n-grams are available. The probabilities estimated from such few n-grams are not reliable. This is why Lingua makes use of n-grams of sizes 1 up to 5 which results in much more accurate prediction of the correct language.

A second important difference is that Lingua does not only use such a statistical model, but also a rule-based engine. This engine first determines the alphabet of the input text and searches for characters which are unique in one or more languages. If exactly one language can be reliably chosen this way, the statistical model is not necessary anymore. In any case, the rule-based engine filters out languages that do not satisfy the conditions of the input text. Only then, in a second step, the probabilistic n-gram model is taken into consideration. This makes sense because loading less language models means less memory consumption and better runtime performance.

In general, it is always a good idea to restrict the set of languages to be considered in the classification process using the respective api methods. If you know beforehand that certain languages are never to occur in an input text, do not let those take part in the classifcation process. The filtering mechanism of the rule-based engine is quite good, however, filtering based on your own knowledge of the input text is always preferable.

8. Test report generation

If you want to reproduce the accuracy results above, you can generate the test reports yourself for all classifiers and languages by executing:

poetry install --only script
poetry run python3 scripts/accuracy_reporter.py

For each detector and language, a test report file is then written into /accuracy-reports. As an example, here is the current output of the Lingua German report:

##### German #####

>>> Accuracy on average: 89.27%

>> Detection of 1000 single words (average length: 9 chars)
Accuracy: 74.20%
Erroneously classified as Dutch: 2.30%, Danish: 2.20%, English: 2.20%, Latin: 1.80%, Bokmal: 1.60%, Italian: 1.30%, Basque: 1.20%, Esperanto: 1.20%, French: 1.20%, Swedish: 0.90%, Afrikaans: 0.70%, Finnish: 0.60%, Nynorsk: 0.60%, Portuguese: 0.60%, Yoruba: 0.60%, Sotho: 0.50%, Tsonga: 0.50%, Welsh: 0.50%, Estonian: 0.40%, Irish: 0.40%, Polish: 0.40%, Spanish: 0.40%, Tswana: 0.40%, Albanian: 0.30%, Icelandic: 0.30%, Tagalog: 0.30%, Bosnian: 0.20%, Catalan: 0.20%, Croatian: 0.20%, Indonesian: 0.20%, Lithuanian: 0.20%, Romanian: 0.20%, Swahili: 0.20%, Zulu: 0.20%, Latvian: 0.10%, Malay: 0.10%, Maori: 0.10%, Slovak: 0.10%, Slovene: 0.10%, Somali: 0.10%, Turkish: 0.10%, Xhosa: 0.10%

>> Detection of 1000 word pairs (average length: 18 chars)
Accuracy: 93.90%
Erroneously classified as Dutch: 0.90%, Latin: 0.90%, English: 0.70%, Swedish: 0.60%, Danish: 0.50%, French: 0.40%, Bokmal: 0.30%, Irish: 0.20%, Tagalog: 0.20%, Tsonga: 0.20%, Afrikaans: 0.10%, Esperanto: 0.10%, Estonian: 0.10%, Finnish: 0.10%, Italian: 0.10%, Maori: 0.10%, Nynorsk: 0.10%, Somali: 0.10%, Swahili: 0.10%, Turkish: 0.10%, Welsh: 0.10%, Zulu: 0.10%

>> Detection of 1000 sentences (average length: 111 chars)
Accuracy: 99.70%
Erroneously classified as Dutch: 0.20%, Latin: 0.10%

9. How to add it to your project?

Lingua is available in the Python Package Index and can be installed with:

pip install lingua-language-detector==1.3.5

10. How to build?

Lingua requires Python >= 3.8 and uses Poetry for packaging and dependency management. You need to install it first if you have not done so yet. Afterwards, clone the repository and install the project dependencies:

git clone https://github.com/pemistahl/lingua-py.git
cd lingua-py
poetry install

The library makes uses of type annotations which allow for static type checking with Mypy. Run the following command for checking the types:

poetry run mypy

The source code is accompanied by an extensive unit test suite. To run the tests, simply say:

poetry run pytest

11. How to use?

11.1 Basic usage

>>> from lingua import Language, LanguageDetectorBuilder
>>> languages = [Language.ENGLISH, Language.FRENCH, Language.GERMAN, Language.SPANISH]
>>> detector = LanguageDetectorBuilder.from_languages(*languages).build()
>>> language = detector.detect_language_of("languages are awesome")
>>> language
Language.ENGLISH
>>> language.iso_code_639_1
IsoCode639_1.EN
>>> language.iso_code_639_1.name
'EN'
>>> language.iso_code_639_3
IsoCode639_3.ENG
>>> language.iso_code_639_3.name
'ENG'

11.2 Minimum relative distance

By default, Lingua returns the most likely language for a given input text. However, there are certain words that are spelled the same in more than one language. The word prologue, for instance, is both a valid English and French word. Lingua would output either English or French which might be wrong in the given context. For cases like that, it is possible to specify a minimum relative distance that the logarithmized and summed up probabilities for each possible language have to satisfy. It can be stated in the following way:

>>> from lingua import Language, LanguageDetectorBuilder
>>> languages = [Language.ENGLISH, Language.FRENCH, Language.GERMAN, Language.SPANISH]
>>> detector = LanguageDetectorBuilder.from_languages(*languages)\
.with_minimum_relative_distance(0.9)\
.build()
>>> print(detector.detect_language_of("languages are awesome"))
None

Be aware that the distance between the language probabilities is dependent on the length of the input text. The longer the input text, the larger the distance between the languages. So if you want to classify very short text phrases, do not set the minimum relative distance too high. Otherwise, None will be returned most of the time as in the example above. This is the return value for cases where language detection is not reliably possible.

11.3 Confidence values

Knowing about the most likely language is nice but how reliable is the computed likelihood? And how less likely are the other examined languages in comparison to the most likely one? These questions can be answered as well:

>>> from lingua import Language, LanguageDetectorBuilder
>>> languages = [Language.ENGLISH, Language.FRENCH, Language.GERMAN, Language.SPANISH]
>>> detector = LanguageDetectorBuilder.from_languages(*languages).build()
>>> confidence_values = detector.compute_language_confidence_values("languages are awesome")
>>> for language, value in confidence_values:
...     print(f"{language.name}: {value:.2f}")
ENGLISH: 0.93
FRENCH: 0.04
GERMAN: 0.02
SPANISH: 0.01

In the example above, a list is returned containing those languages which the calling instance of LanguageDetector has been built from, sorted by their confidence value in descending order. Each value is a probability between 0.0 and 1.0. The probabilities of all languages will sum to 1.0. If the language is unambiguously identified by the rule engine, the value 1.0 will always be returned for this language. The other languages will receive a value of 0.0.

There is also a method for returning the confidence value for one specific language only:

>>> from lingua import Language, LanguageDetectorBuilder
>>> languages = [Language.ENGLISH, Language.FRENCH, Language.GERMAN, Language.SPANISH]
>>> detector = LanguageDetectorBuilder.from_languages(*languages).build()
>>> confidence_value = detector.compute_language_confidence("languages are awesome", Language.FRENCH)
>>> print(f"{confidence_value:.2f}")
0.04

The value that this method computes is a number between 0.0 and 1.0. If the language is unambiguously identified by the rule engine, the value 1.0 will always be returned. If the given language is not supported by this detector instance, the value 0.0 will always be returned.

11.4 Eager loading versus lazy loading

By default, Lingua uses lazy-loading to load only those language models on demand which are considered relevant by the rule-based filter engine. For web services, for instance, it is rather beneficial to preload all language models into memory to avoid unexpected latency while waiting for the service response. If you want to enable the eager-loading mode, you can do it like this:

LanguageDetectorBuilder.from_all_languages().with_preloaded_language_models().build()

Multiple instances of LanguageDetector share the same language models in memory which are accessed asynchronously by the instances.

11.5 Low accuracy mode versus high accuracy mode

Lingua's high detection accuracy comes at the cost of being noticeably slower than other language detectors. The large language models also consume significant amounts of memory. These requirements might not be feasible for systems running low on resources. If you want to classify mostly long texts or need to save resources, you can enable a low accuracy mode that loads only a small subset of the language models into memory:

LanguageDetectorBuilder.from_all_languages().with_low_accuracy_mode().build()

The downside of this approach is that detection accuracy for short texts consisting of less than 120 characters will drop significantly. However, detection accuracy for texts which are longer than 120 characters will remain mostly unaffected.

In high accuracy mode (the default), the language detector consumes approximately 800 MB of memory if all language models are loaded. In low accuracy mode, memory consumption is reduced to approximately 60 MB.

An alternative for a smaller memory footprint and faster performance is to reduce the set of languages when building the language detector. In most cases, it is not advisable to build the detector from all supported languages. When you have knowledge about the texts you want to classify you can almost always rule out certain languages as impossible or unlikely to occur.

11.6 Detection of multiple languages in mixed-language texts

In contrast to most other language detectors, Lingua is able to detect multiple languages in mixed-language texts. This feature can yield quite reasonable results but it is still in an experimental state and therefore the detection result is highly dependent on the input text. It works best in high-accuracy mode with multiple long words for each language. The shorter the phrases and their words are, the less accurate are the results. Reducing the set of languages when building the language detector can also improve accuracy for this task if the languages occurring in the text are equal to the languages supported by the respective language detector instance.

>>> from lingua import Language, LanguageDetectorBuilder
>>> languages = [Language.ENGLISH, Language.FRENCH, Language.GERMAN]
>>> detector = LanguageDetectorBuilder.from_languages(*languages).build()
>>> sentence = "Parlez-vous français? " + \
...            "Ich spreche Französisch nur ein bisschen. " + \
...            "A little bit is better than nothing."
>>> for result in detector.detect_multiple_languages_of(sentence):
...     print(f"{result.language.name}: '{sentence[result.start_index:result.end_index]}'")
FRENCH: 'Parlez-vous français? '
GERMAN: 'Ich spreche Französisch nur ein bisschen. '
ENGLISH: 'A little bit is better than nothing.'

In the example above, a list of DetectionResult is returned. Each entry in the list describes a contiguous single-language text section, providing start and end indices of the respective substring.

11.7 Methods to build the LanguageDetector

There might be classification tasks where you know beforehand that your language data is definitely not written in Latin, for instance. The detection accuracy can become better in such cases if you exclude certain languages from the decision process or just explicitly include relevant languages:

from lingua import LanguageDetectorBuilder, Language, IsoCode639_1, IsoCode639_3

# Include all languages available in the library.
LanguageDetectorBuilder.from_all_languages()

# Include only languages that are not yet extinct (= currently excludes Latin).
LanguageDetectorBuilder.from_all_spoken_languages()

# Include only languages written with Cyrillic script.
LanguageDetectorBuilder.from_all_languages_with_cyrillic_script()

# Exclude only the Spanish language from the decision algorithm.
LanguageDetectorBuilder.from_all_languages_without(Language.SPANISH)

# Only decide between English and German.
LanguageDetectorBuilder.from_languages(Language.ENGLISH, Language.GERMAN)

# Select languages by ISO 639-1 code.
LanguageDetectorBuilder.from_iso_codes_639_1(IsoCode639_1.EN, IsoCode639_1.DE)

# Select languages by ISO 639-3 code.
LanguageDetectorBuilder.from_iso_codes_639_3(IsoCode639_3.ENG, IsoCode639_3.DEU)

12. Performance tips

If you find that Lingua is too slow for your purposes, e.g. when you classify a large amount of texts, a sensible option is to make use of process-based parallelism via the multiprocessing module. This allows to distribute the workload over multiple CPU cores which improves speed of execution at the cost of higher memory consumption.

The easiest way to implement it is by making use of ProcessPoolExecutor:

from concurrent.futures import ProcessPoolExecutor
from lingua import LanguageDetectorBuilder

detector = (
  LanguageDetectorBuilder.from_all_languages()
  .with_preloaded_language_models()
  .build()
)

def classify(text):
    return detector.detect_language_of(text)

n = 4 # Set n to the number of CPU cores in your machine
texts = ["huge amount", "of texts", ...]

with ProcessPoolExecutor(max_workers=n) as executor:
    results = executor.map(classify, texts)
    for detected_language in results:
        # process the results
        ...

Be aware that starting n processes will consume n times more memory for the language models. In a single process, all language models consume around 3 GB of memory. So when starting 4 processes, memory consumption will increase up to 12 GB!

13. What's next for version 1.4.0?

Take a look at the planned issues.

14. Contributions

Any contributions to Lingua are very much appreciated. Please read the instructions in CONTRIBUTING.md for how to add new languages to the library.