langfamilysim

Estimate how similar languages are, three ways:

• langfamilysim: genealogical similarity from Glottolog's classification tree, with a geographic-distance damped variant. Low-resource language in, closest related languages out.
• lwc: for a language, which languages of wider communication its speakers probably also speak (a directional bilingualism probability, e.g. Isanzu → Swahili), from Glottolog and Unicode CLDR data.
• biblesim: empirical similarity from verse-aligned Bibles in the eBible corpus, comparing words (Levenshtein), cognate rate, verse length and word order.

Requires Python 3.12+.

pip install langfamilysim

Quick start

Languages of wider communication (works out of the box)

from lwc.lookup import LWCLookup

lwc = LWCLookup()
for x in lwc.likely_lwcs("isn", k=3):      # ISO 639-3 or glottocode
    print(x.name, x.probability, x.country)   # Swahili 0.9 TZ, English 0.69 TZ

Genealogical similarity (build the graph first)

The full graph (similarity.db, about 290 MB) is too large for PyPI, so you build it from Glottolog. From a source checkout:

git clone https://github.com/TangoGroup/language-similarity.git
cd language-similarity/src/langfamilysim
./fetch_glottolog.sh      # sparse clone of the Glottolog tree, under a minute
python3 build.py          # writes nodes.csv, edges.csv, similarity.db (~20 s)

from similarity import LanguageSimilarity, name_lookup

names, iso_to_gc = name_lookup()
sim = LanguageSimilarity()
for other, score, kind in sim.top_related(iso_to_gc["spa"], k=10, by="combined"):
    print(names[other], score, kind)

Each edge has a tree-based score and a combined score damped by geographic distance (by="score" ignores geography). See src/langfamilysim/README.md for the method.

Bible-based similarity

Download the eBible corpus separately (it is not included), then:

export BIBLESIM_CORPUS=/path/to/ebible/corpus
python -m biblesim compare spa por
python -m biblesim --stride 10 family indo1319 --langs spa,por,fra,ita,ron
python -m biblesim validate        # Spearman correlation vs the Glottolog scores

Language families come from the similarity.db built above (--langdb). See src/biblesim/README.md.

Data and licensing

The code is Apache-2.0. The data it ships or derives comes from third parties under their own licenses: Glottolog and WALS (CC BY 4.0) and Unicode CLDR (Unicode License). eBible translations are licensed individually, so check each before redistributing any text. Details, credit lines and citations are in DATA_SOURCES.md.

Geographic data is approximate: Glottolog gives one point per language, so distances are only a rough proxy for regional multilingualism.

Development

uv sync
uv run pytest

License

Apache License 2.0. See LICENSE and NOTICE.