languages-of-the-world 0.1.0

Languages of the World — an in-memory object graph for language, country, continent, and regional data

low — Languages of the World

low is a lightweight, read-only Python utility that aggregates and normalises global language, country, continent, regional, and per-country speaker count data into a connected in-memory object graph.

Instead of wrapping data behind traditional repository classes, low exposes everything through idiomatic Python sequences, smart multi-key lookups, and direct dot-notation object navigation.

---

Installation

pip install low

---

Quick Start

import low

# Initialise the graph (~7900 languages, 247 countries, 4800+ family tree nodes)
db = low.LanguagesOfTheWorld()

# Slice like a list
first_ten = db.languages[:10]
total     = len(db.languages)

# Polymorphic single-key lookup
db.languages.get("rw")          # ISO 639-1 (2-char)  → Kinyarwanda
db.languages.get("kin")         # ISO 639-3 (3-char)  → Kinyarwanda
db.languages.get("Kinyarwanda") # Label (case-insensitive) → Kinyarwanda

# Navigate the object graph with dot notation
lang = db.languages.get("kin")
for country in lang.countries:
    print(f"{country.label} — {country.region.label} ({country.continent.label})")
# Rwanda — Eastern Africa (Africa)
# DR Congo — Eastern Africa (Africa)
# Uganda — Eastern Africa (Africa)

# Country → back-reference to languages
rw = db.countries.get("RW")
for l in rw.languages:
    print(l.label)

# Filter by partial name or minimum speakers
popular = db.languages.filter(min_speakers=50_000_000)
romance = db.languages.filter(label_contains="Portug")

# Glottolog endangerment status
print(db.languages.get("kin").endangerment)   # 'not_endangered'
print(db.languages.get("dlg").endangerment)   # 'moribund'  (Dolgan)
at_risk = [l for l in db.languages
           if l.endangerment in {"nearly_extinct", "moribund"}]

# Walk the Glottolog language family tree
deu = db.languages.get("deu")
node = deu.family
while node:
    print("  " * node.depth + node.label)
    node = node.parent
# Standard German's Glottolog lineage, leaf → root

# All top-level root families
for fam in db.families.roots():
    print(fam.label, f"({len(fam.children)} subgroups)")

# Official language status per country (from CLDR)
ch = db.countries.get("CH")
print([l.label for l in ch.official_languages])
# ['French', 'German', 'Italian', 'Romansh']
print([l.label for l in ch.official_regional_languages])
# e.g. regionally recognised languages
print([l.label for l in ch.de_facto_official_languages])
# e.g. de facto official languages with no formal legal status

# Per-country speaker counts — how many people speak a language in each country
rw = db.countries.get("RW")
print(f"Rwanda population: {rw.population:,}")
for sc in rw.speaker_counts:
    print(f"{sc.language.label}: {sc.speaker_count:,} ({sc.speaker_fraction:.1%}) [{sc.source}]")
# Rwanda population: 13,776,698
# Kinyarwanda: 10,200,000 (74.0%) [cldr]
# French: 300,000 (2.2%) [cldr]
# Kinyarwanda: 9,900,000 (71.8%) [cia]

# Same from the language side
kin = db.languages.get("kin")
for sc in kin.speaker_counts:
    print(f"{sc.country.label}: {sc.speaker_count:,} ({sc.source})")

# Canonical names for a language across other languages
deu = db.languages.get("deu")
print(deu.endonym.name)                       # "Deutsch"
print([n.name for n in deu.names if n.in_language_bcp47 == "fr"])
# ['allemand']

# All known English names for every language
for n in db.language_names.in_language("en")[:5]:
    print(f"{n.language.part3} → {n.name}")

# Query the full SpeakerCount collection directly
db.speaker_counts.for_country("DE")        # all entries for Germany
db.speaker_counts.for_language("deu")      # all entries for German
db.speaker_counts.by_source("cldr")        # all CLDR-sourced entries
db.speaker_counts.by_source("cia")         # all CIA-sourced entries
db.speaker_counts.by_source("linguameta")  # all LinguaMeta-sourced entries

---

Entity Model

[Continent] <───1:N─── [Region] <───1:N─── [Country] <───M:N─── [Language] ───1:N─── [LanguageName]
     │                                           │                     │                    │
     └───────────────────1:N────────────────────┘                     └───N:1─── [LanguageFamily]
                                │                                                      │ parent/children
                                └───────────── [SpeakerCount] ─────────────────────────
                                                  (country, language,
                                                   speaker_count, source)

Language

Property	Type	Source	Description
part3	str	SIL ISO 639-3	ISO 639-3 three-letter code — primary key
part1	Optional[str]	SIL ISO 639-3	ISO 639-1 two-letter code (if assigned)
label	str	SIL ISO 639-3	Reference name
scope	str	SIL ISO 639-3	"I" Individual · "M" Macrolanguage · "S" Special
speaker_count	int	LinguaMeta / Wikidata	Estimated total speakers (global), max across sources
countries	List[Country]	LinguaMeta	Countries where the language is spoken
family	Optional[LanguageFamily]	Glottolog	Immediate parent node in the Glottolog tree
glottocode	Optional[str]	Glottolog	Glottolog identifier (e.g. "kin1248")
endangerment	Optional[str]	Glottolog	Agglomerated Endangerment Status (AES). One of "not_endangered", "threatened", "shifting", "moribund", "nearly_extinct", "extinct"; None if Glottolog has no assessment
speaker_counts	List[SpeakerCount]	CLDR / CIA / LinguaMeta	Per-country speaker counts for this language
names	List[LanguageName]	LinguaMeta	Canonical names for this language in other languages
endonym	Optional[LanguageName] (property)	LinguaMeta	The name expressed in the language itself, if available

LanguageName

A single canonical name for a language, expressed in some (possibly different) language. Sourced from LinguaMeta's name_data, filtered to is_canonical=True.

Property	Type	Source	Description
language	Language	—	The language being named
name	str	LinguaMeta	The name string (e.g. "Deutsch", "German", "Allemand")
in_language_bcp47	str	LinguaMeta	BCP 47 code of the language the name is expressed in
in_language	Optional[Language]	derived	Resolved Language, when the BCP 47 base maps to a known ISO 639-3
script	Optional[str]	LinguaMeta	ISO 15924 script code, when supplied
source	Optional[str]	LinguaMeta	Upstream provenance string (e.g. "CLDR", "GOOGLE_RESEARCH")
is_endonym	bool (property)	derived	True when in_language is the same as language

Country

Property	Type	Source	Description
code	str	UN M49	ISO 3166-1 alpha-2
label	str	UN M49	Common name
continent	Continent	UN M49
region	Region	UN M49	UN M49 sub-region
population	int	CLDR	Total population (0 if not available)
languages	List[Language]	LinguaMeta	All languages spoken in this country
official_languages	List[Language]	CLDR	Nationally official languages (officialStatus="official")
official_regional_languages	List[Language]	CLDR	Regionally official languages (officialStatus="official_regional")
de_facto_official_languages	List[Language]	CLDR	De facto official languages (officialStatus="de_facto_official")
speaker_counts	List[SpeakerCount]	CLDR / CIA / LinguaMeta	Per-language speaker counts in this country

SpeakerCount

Represents how many speakers of a given language live in a given country, according to a specific data source. Both country.speaker_counts and language.speaker_counts navigate to these objects.

Property	Type	Source	Description
country	Country	—	The country
language	Language	—	The language
speaker_count	int	CLDR / CIA / LinguaMeta	Estimated number of speakers
speaker_fraction	float	CLDR / CIA / LinguaMeta	Share of country population (0.0–1.0; derived from Country.population for LinguaMeta)
source	str	—	"cldr", "cia", or "linguameta"

Region

Property	Type	Source	Description
id	str	UN M49	UN M49 numeric code
label	str	UN M49	Sub-region name
continent	Continent	UN M49
countries	List[Country]	UN M49	Back-reference

Continent

Property	Type	Source	Description
id	str	UN M49	UN M49 numeric code
label	str	UN M49	Continent name
countries	List[Country]	UN M49	Back-reference

LanguageFamily

Represents a node in the Glottolog genealogical classification tree. Every node — from the deepest sub-branch to a top-level family like Indo-European — is a LanguageFamily instance.

Property	Type	Source	Description
glottocode	str	Glottolog	Glottolog identifier (e.g. "indo1319")
label	str	Glottolog	Node name
parent	Optional[LanguageFamily]	Glottolog	Parent node; None for root families
children	List[LanguageFamily]	Glottolog	Direct child sub-families
languages	List[Language]	Glottolog	Languages whose immediate Glottolog parent is this node
root	LanguageFamily (property)	derived	Walk up to the top-level family
ancestors	List[LanguageFamily] (property)	derived	Ordered list parent → root
depth	int (property)	derived	Depth in tree (0 = root family)

---

Collection Interface

Every entity collection (db.languages, db.countries, db.continents, db.regions, db.families, db.speaker_counts, db.language_names) implements the Python sequence protocol:

len(db.languages)        # int
db.languages[0]          # Language
db.languages[:5]         # List[Language]
for lang in db.languages: ...

.get(query) — Polymorphic lookup

Query pattern	Resolves to
2-char string	ISO 639-1 / ISO 3166-1 alpha-2
3-char string	ISO 639-3
8-char string (families)	Glottolog code
Longer string	Case-insensitive label

.filter() (LanguageCollection only)

db.languages.filter(label_contains="arabic")
db.languages.filter(min_speakers=1_000_000)
db.languages.filter(label_contains="creole", min_speakers=100_000)

.roots() (FamilyCollection only)

db.families.roots()   # List[LanguageFamily] — only top-level families

SpeakerCountCollection (db.speaker_counts)

The SpeakerCount collection adds three targeted query methods on top of the standard sequence protocol:

db.speaker_counts.for_country("DE")    # List[SpeakerCount] — all entries for Germany
db.speaker_counts.for_language("deu")  # List[SpeakerCount] — all entries for German
db.speaker_counts.by_source("cldr")        # List[SpeakerCount] — CLDR entries only
db.speaker_counts.by_source("cia")         # List[SpeakerCount] — CIA entries only
db.speaker_counts.by_source("linguameta")  # List[SpeakerCount] — LinguaMeta entries only

LanguageNameCollection (db.language_names)

Holds every canonical name parsed from LinguaMeta's per-language JSON (name_data rows with is_canonical=True), one record per (language, in_language_bcp47, script) triple.

db.language_names.for_language("deu")   # every canonical name of German
db.language_names.in_language("en")     # every language's English canonical name
db.language_names.endonyms()            # one entry per language: its name in itself

Each entry resolves to a Language via name.in_language when the BCP 47 base maps to a known ISO 639-3; otherwise in_language is None and only in_language_bcp47 is meaningful.

The same data is also reachable through dot navigation on Country and Language:

db.countries.get("DE").speaker_counts   # identical to for_country("DE")
db.languages.get("deu").speaker_counts  # identical to for_language("deu")

---

Data Provenance

low integrates six open datasets. The bootstrap pipeline (python -m low.bootstrap) fetches them at build time and bakes the result into src/low/data/low_db.json.

SIL International — ISO 639-3

URL: https://iso639-3.sil.org/sites/iso639-3/files/downloads/iso-639-3.tab
Licence: SIL Usage
Fields provided:

Field	Column in source
Language.part3	Id
Language.part1	Part1
Language.label	Ref_Name
Language.scope	Scope (I/M/S)

This TSV is the authoritative source of all ISO 639-3 codes. Every language in low originates here; other sources add extra attributes.

---

UN M49 — ISO-3166-Countries-with-Regional-Codes

URL: https://raw.githubusercontent.com/lukes/ISO-3166-Countries-with-Regional-Codes/master/all/all.csv
Licence: MIT
Fields provided:

Field	Column in source
Country.code	alpha-2
Country.label	name
Region.id, Region.label	sub-region-code, sub-region
Continent.id, Continent.label	region-code, region

Countries, regions, and continents are entirely built from this CSV.

---

Google Research — LinguaMeta

TSV URL: https://raw.githubusercontent.com/google-research/url-nlp/main/linguameta/linguameta.tsv Per-language JSON base: https://raw.githubusercontent.com/google-research/url-nlp/main/linguameta/data/<bcp47>.json Licence: CC BY 4.0 Raw output: src/low/data/sources/linguameta_speakers.json Fields provided:

Field	Source path
Language.speaker_count	TSV estimated_number_of_speakers (merged max with Wikidata)
Language.countries	TSV locales (comma-separated ISO 3166-1 alpha-2)
SpeakerCount.country	per-language JSON language_script_locale[].locale.iso_3166_code
SpeakerCount.language	per-language JSON iso_639_3_code
SpeakerCount.speaker_count	language_script_locale[].speaker_data.number_of_speakers
SpeakerCount.speaker_fraction	derived as speaker_count / Country.population (CLDR-sourced population)
LanguageName.name	per-language JSON name_data[].name (filtered to is_canonical=true)
LanguageName.in_language_bcp47	name_data[].bcp_47_code
LanguageName.script	name_data[].iso_15924_code (optional)
LanguageName.source	name_data[].source (e.g. "CLDR", "GOOGLE_RESEARCH")

The TSV is the authoritative table for the global per-language total (estimated_number_of_speakers, order-of-magnitude rounded). Multiple BCP-47 rows mapping to the same ISO 639-3 code are merged (max speakers, union of country codes).

The per-language JSON files under linguameta/data/ are fetched in parallel (~7 000 files, up to 20 threads) in a single pass that produces two record sets:

• Per-locale speaker counts from language_script_locale[].speaker_data.number_of_speakers → merged into SpeakerCount with source="linguameta".
• Canonical names from name_data[] (rows with is_canonical=true) → LanguageName collection. Names are deduplicated on (language, in_language_bcp47, script); the first occurrence wins.

The repo file tree is discovered via a single GitHub trees API call; individual files come from raw.githubusercontent.com (no rate-limit).

---

Glottolog CLDF

Languages CSV: https://raw.githubusercontent.com/glottolog/glottolog-cldf/master/cldf/languages.csv
Values CSV: https://raw.githubusercontent.com/glottolog/glottolog-cldf/master/cldf/values.csv
Licence: CC BY 4.0
Fields provided:

Field	Source column
LanguageFamily.glottocode	ID (languages.csv, Level=family)
LanguageFamily.label	Name (languages.csv)
LanguageFamily.parent	Last component of the classification path (values.csv)
Language.glottocode	ID (languages.csv, Level=language)
Language.family	Resolved from the language's classification path
Language.endangerment	Code_ID of the aes parameter (values.csv), with the aes- prefix stripped

The classification value is a slash-separated ancestor chain from the root family down to the node's immediate parent (e.g. "indo1319/clas1257/germ1287/nort3152/west2793/high1289/high1286/midd1349/mode1258/uppe1464/glob1243"). The bootstrap extracts the last element as the immediate parent, producing the full navigable tree. The production database contains ~4 800 family-level nodes and 246 root families.

---

Unicode CLDR — supplementalData.xml

URL: https://raw.githubusercontent.com/unicode-org/cldr/main/common/supplemental/supplementalData.xml
Licence: Unicode License v3
Raw output: src/low/data/sources/cldr_speakers.json
Fields provided:

Field	Source element
Country.population	territory[@population]
Country.official_languages	languagePopulation[@officialStatus="official"]
Country.official_regional_languages	languagePopulation[@officialStatus="official_regional"]
Country.de_facto_official_languages	languagePopulation[@officialStatus="de_facto_official"]
SpeakerCount.country	<territory type="…"> — ISO 3166-1 alpha-2
SpeakerCount.language	<languagePopulation type="…"> — BCP 47 tag, mapped to ISO 639-3
SpeakerCount.speaker_count	territory[@population] × languagePopulation[@populationPercent] / 100
SpeakerCount.speaker_fraction	languagePopulation[@populationPercent] / 100

Each <territory> element carries the total population and a list of <languagePopulation> children with populationPercent and optional officialStatus attributes. The three official-status lists on Country are built from the officialStatus attribute; entries with any other value (e.g. official_minority) or no attribute are omitted from those lists. Territory population is written directly onto Country.population. BCP 47 language tags are normalised to ISO 639-3 codes using the SIL table (2-char base → ISO 639-1 → ISO 639-3 mapping; 3-char base used directly).

---

CIA World Factbook — factbook.json

Index URL: https://raw.githubusercontent.com/factbook/factbook.json/master/index.json
Per-country base: https://raw.githubusercontent.com/factbook/factbook.json/master/
Licence: Public domain (U.S. Government work)
Raw output: src/low/data/sources/cia_speakers.json
Fields provided:

Field	Source path in country JSON
SpeakerCount.country	Government › Country name › conventional short form — matched to ISO alpha-2 by label
SpeakerCount.language	People and Society › Languages › language[].name — matched to ISO 639-3 by label
SpeakerCount.speaker_count	country population × language percent / 100
SpeakerCount.speaker_fraction	language percent / 100

Country files are fetched in parallel (up to 20 threads). Each file's language section is parsed as a structured list when available or extracted from free text via a percentage regex as fallback. Language names are matched to ISO 639-3 codes against the Language.label index.

---

Wikidata — SPARQL Query Service

Endpoint: https://query.wikidata.org/sparql Licence: CC0 1.0 Raw output: src/low/data/sources/wikidata_speakers.json Fields provided:

Field	Source property
Language.speaker_count	P1098 (number of speakers, writers, or signers), max with LinguaMeta

Queries all instances of wd:Q34770 (language) and its subclasses that carry a P1098 value, joined with P218 (ISO 639-1) and P220 (ISO 639-3) for code resolution. Rows resolve to ISO 639-3 directly when present, otherwise via the 2-letter code lookup. When multiple Wikidata items map to the same ISO 639-3 (sub-varieties, alternative references) the highest speaker count wins. Merged into Language.speaker_count as max(linguameta, wikidata).

---

Regenerating the Database

The baked JSON (src/low/data/low_db.json) is shipped with the package. To re-pull from upstream sources (requires internet access):

pip install "low[bootstrap]"
python -m low.bootstrap

The bootstrap writes three files:

File	Contents
src/low/data/low_db.json	Merged, deduplicated graph database (includes country_official_languages section)
src/low/data/sources/cldr_speakers.json	Raw CLDR per-territory language population records (includes official_status field)
src/low/data/sources/cia_speakers.json	Raw CIA World Factbook per-country language records
src/low/data/sources/linguameta_speakers.json	Raw LinguaMeta per-locale speaker-count records
src/low/data/sources/linguameta_names.json	Raw LinguaMeta canonical language-name records
src/low/data/sources/wikidata_speakers.json	Raw Wikidata SPARQL global speaker-count records

The two source files preserve the original data exactly as parsed, before deduplication and ISO-code resolution, so they can be used independently.

---

Examples

See the examples/ directory for Jupyter notebooks:

Notebook	Description
01_languages_per_country.ipynb	World choropleth map — number of languages per country

---

Development

git clone https://github.com/your-org/low
cd low
pip install -e ".[dev]"
pytest

---

License

MIT