Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

linguatag

BCP-47 language tags: standardization, distance, closest match, and the default region of a tag that names none.

from linguatag import (closest_lang, default_region, is_arabic_variety, lang_distance,
                       with_default_region)

default_region("en")            # 'GB'
with_default_region("zh-Hant")  # 'zh-Hant-TW'
lang_distance("pt", "pt-PT")    # 0
lang_distance("pt", "pt-BR")    # 5
closest_lang("en", ["en-US", "en-GB"])  # 'en-GB'
closest_lang("en-AU", ["fr-FR", "en-US"])  # 'en-US'
is_arabic_variety("ar-Latn-x-buckwalt")  # True
is_arabic_variety("fa")         # False

Default region

A tag with no region defaults to its motherland: the region where the language first appeared and is still spoken. en is en-GB, pt is pt-PT, es is es-ES.

CLDR likely-subtags is the base. CLDR picks the most-populous region, and for most languages that is also the motherland. linguatag.OVERRIDES replaces CLDR where the two differ, and each override cites its source: en is GB, pt is PT, ur is IN and yue is CN. An override holds for its own script only, so yue-Hant stays HK. When several countries share the area where a language first appeared, the CLDR pick is one of them and stays.

A bare ar is Modern Standard Arabic, the standard of no single country, so its region is 001, the world, as CLDR spells it.

linguatag/data/likely_regions.json holds the CLDR regions. scripts/build_likely_regions.py rebuilds it from the installed langcodes.

The CLDR data in langcodes has no entry for some Arabic varieties, and it puts them in Latin script. CLDR 49 puts them in Arabic script: Libyan Arabic ayl is ayl-Arab-LY and Hassaniyya mey is mey-Arab-DZ. For these varieties the table holds the CLDR 49 entries, with their scripts, and the build script reads them from the CLDR release. A tag that names no script is measured in the CLDR script, so mey and mey-Arab are at distance 0. Hassaniyya in Senegal, mey-SN, is in Latin script, as CLDR gives it.

The same holds for two more languages. The langcodes data has no entry for Western Armenian and puts it in Latin script in the US, and it puts Literary Chinese in Simplified script. CLDR 49 gives hyw_Armn_AM and lzh_Hant_CN, and the table holds these entries. So hyw is hyw-Armn-AM and lzh-TW is lzh-Hant-TW. A hyw-Latn request matches no Armenian-script resource, such as hyw or hyw-AM.

Distance and closest match

lang_distance is 0 for identical tags and grows as tags differ. A distance of 10 or less is a usable match. Both tags are standardized first, and a tag with no region is measured from its default region.

Standardization replaces an extlang tag and a deprecated language subtag with their IANA Preferred-Values, as RFC 5646 section 4.5 says: zh-hak is Hakka (hak), ar-arz-EG is arz-EG, and ajp-JO is apc-JO. linguatag/data/extlangs.json and linguatag/data/deprecated_languages.json hold the tables. langcodes replaces a deprecated region subtag (my-BU is my-MM), and linguatag/data/deprecated_regions.json holds the IANA Preferred-Values of region subtags for the tie rule below. scripts/build_preferred_values.py rebuilds the three tables from the IANA Language Subtag Registry.

closest_lang returns the candidate with the smallest distance, verbatim, when that distance is within max_distance (default 10). Pass max_distance=0 to accept exact matches only. Among candidates at the same distance, the code that the request names wins first, and a code with no language, extlang or region subtag that has a Preferred-Value wins next: closest_lang("zh-hak", ["hak", "zh-hak"]) is zh-hak, closest_lang("ar-apc", ["ajp", "apc"]) is apc, and closest_lang("my-MM", ["my-BU", "my-MM"]) is my-MM.

An Arabic variety (arz, apc, ar-MA, Hassaniyya mey) in Arabic script is at most 5 from Modern Standard Arabic (ar, arb). Juba Arabic (pga), Cypriot Maronite Arabic (acy) and Hassaniyya in Senegal (mey-SN) are written in Latin script, so they get this rule only with an explicit -Arab subtag. No other macrolanguage gets this rule. No spoken member of the zh macrolanguage other than Mandarin matches cmn or zh in Simplified script: Cantonese, Hakka, Min Nan, Wu, Gan, Xiang, Min Dong, Min Bei, Jinyu and the others the IANA registry lists under zh. zh in Traditional script and zh-Latn keep the langcodes distance for them. Literary Chinese (lzh) has no speakers, so this rule leaves it out. CLDR 49 writes it in Traditional script in China, so a Chinese resource in the same script and region serves an lzh request: lzh gets zh-Hant-CN, lzh-TW gets zh-Hant, and lzh-Hans gets zh or cmn. A bare lzh is 14 from zh-Hant, which is Taiwan, because the regions differ. It does not match Simplified Chinese. Fanti (fat) and Mandinka (mnk) match no other language, Akan (ak), Twi (tw), Mandingo (man) and Eastern Maninkakan (emk) included: CLDR 49 does not make them aliases of a macrolanguage. Twi is an alias of Akan.

Arabic varieties

is_arabic_variety tells whether a tag names Arabic. The language subtag decides. Script, region and private-use subtags do not count, so ar-Latn-x-buckwalt and Juba Arabic (pga) are Arabic varieties. Arabic is ar (ara), the individual languages that the IANA registry puts in the ar macrolanguage (arb, arz, acy, pga and the others), and Hassaniyya (mey). The IANA registry puts Hassaniyya in no macrolanguage, and Glottolog classifies it as Arabic. A different language in Arabic script is not Arabic: fa, ur and und-Arab are False.

lang_distance answers a different question: whether a resource for one tag can serve a request for another.

Dialects

A private-use subtag names a dialect, such as an-x-cheso or pt-PT-x-porto. BCP-47 gives it no meaning, so by default it adds nothing to the distance, and an-x-cheso is as near to an-x-ansotano as to an. Pass lects to score it:

closest_lang("an-x-cheso", ["an", "an-x-ansotano"], lects="dialect")   # 'an-x-ansotano'
closest_lang("an-x-cheso", ["an", "an-x-ansotano"], lects="standard")  # 'an'

"dialect" takes the nearest other dialect first and the standard language last. "standard" takes the standard language first and the dialects after it. Nearness is the number of steps between two dialects in the orthography2ipa dialect tree: a parent dialect is 1 step away, a sibling 2. The lect score adds at most 4. closest_lang uses it only to order the candidates within max_distance, so it never decides whether a tag matches.

linguatag/data/lineage.json holds the tree. scripts/build_lineage.py rebuilds it from the orthography2ipa spec files at a git ref.

Install

pip install linguatag[langcodes]

langcodes is optional. Without it, standardization only folds case and underscores and replaces extlang tags, and distance uses a coarse same-language measure with the same ordering.

Metadata

Release files for linguatag 0.2.7a1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for linguatag 0.2.7a1
File Size Uploaded Provenance Actions
linguatag-0.2.7a1.tar.gz 44.5 kB Without provenance Details for linguatag-0.2.7a1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for linguatag 0.2.7a1
File Interpreter ABI Platform Actions
linguatag-0.2.7a1-py3-none-any.whl Python 3 none any Details for linguatag-0.2.7a1-py3-none-any.whl

Total release size: 75.2 kB

Release files / linguatag-0.2.7a1.tar.gz

Download URL linguatag-0.2.7a1.tar.gz
Size 44.5 kB
Tags Source
SHA-256 checksum
How to use checksums
26b555cd3c0cc390912a586d11cf271aa268e2a924e50b605a27235dc0268984
BLAKE2b-256 checksum
How to use checksums
87b51a426a500719c5f788c6ee1ea61e99fe12ffe3d330f4d3c6e8f3ccac74ab
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / linguatag-0.2.7a1-py3-none-any.whl

Download URL linguatag-0.2.7a1-py3-none-any.whl
Size 30.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a0a5b86e2f9439c14a0af6b1502633f1e2dfbbf3ccc56687bc4514c7fd5b6289
BLAKE2b-256 checksum
How to use checksums
d05458567491f2a68bc142ee146116161eb453fb7a77299c512a996548bed6f7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Security for linguatag 0.2.7a1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page