Qwanqwa (qq): Language Metadata
A unified language metadata toolkit for NLP: identifiers, scripts, speakers, geographic data, and traversable relationships across ~27,315 languoids.
Name: Qwanqwa is a phonetic spelling of 'ቋንቋ', which means language in Amharic; qq is short to type.
Features
- Identifiers: BCP-47, ISO 639-1, ISO 639-3, ISO 639-2B, ISO 639-2T, ISO 639-5, Glottocode, Wikidata ID, Wikipedia ID, NLLB-style codes
- Geographic information: Countries and territories, which can be traversed from languoids and back
- Speaker information: Population counts, UNESCO endangerment status
- Resource availability: Joshi et al. (2020) NLP resource ratings, Hugging Face dataset counts, and Wikipedia coverage
- Writing systems: ISO 15924 codes and names, script types and families, native-script examples, canonical/historical metadata, and Unicode ranges
- Multilingual names: Language names in 500+ languages
- Relationships: Traversable graph of language families, scripts, and geographic regions
- Phylogenetic data: Language family trees from Glottolog
- External resources: Find datasets and reference resources that include a particular language. Currently includes CLLD datasets like WALS and Grambank, Hugging Face Datasets, and Universal Dependencies.
Languoids
In qq, language-like entities are referred to as Languoids: this includes dialects, macro-languages, and language families, not just individual languages. Not all languoids have coverage for all features.
Installation
uv add qwanqwa
# or
pip install qwanqwa
Quick Start
from qq import Database, IdType
# Load the pre-compiled database
db = Database.load()
# Get a language by BCP-47 code (default)
dutch = db.get("nl")
print(dutch.name) # "Dutch"
print(dutch.iso_639_3) # "nld"
print(dutch.speaker_count) # 24085200
print(dutch.joshi_rating) # 4 (on the 0–5 Joshi resource scale)
# Also works with ISO 639-3, Glottocode, etc.
dutch2 = db.get("nld", id_type=IdType.ISO_639_3)
dutch3 = db.get("dutc1256", id_type=IdType.GLOTTOCODE)
dutch4 = db.guess("dut") # guessing works too
# This will all resolve to the same languoid
assert dutch == dutch2 == dutch3 == dutch4
# Search by name
results = db.search("Chinese")
for lang in results:
print(f"{lang.name} ({lang.glottocode})")
# Search also accepts identifiers and ranks them higher
db.search("nl")[0].name # "Dutch"
db.search("English")[0].bcp_47 # "en"
Important: qq makes a strict distinction between None (don't know) and False (it is not the case). When checking boolean attributes, prefer explicit checks over truthiness: use if script.is_canonical is None: rather than if not script.is_canonical:.
Traversal
Languoids, scripts, and geographic regions are all part of the same graph, which can be traversed:
dutch = db.get("nl")
# Language family navigation (Glottolog tree)
dutch.parent # Global Dutch
dutch.parent.parent # Modern Dutch
dutch.family_tree # [Global Dutch, Modern Dutch, ..., West Germanic, Germanic, Indo-European]
dutch.siblings # [Afrikaansic, Javindo, Petjo]
dutch.children # [North Hollandish, Central Northern Dutch, ...]
dutch.descendants() # All descendants (recursive)
# Writing systems
dutch.scripts # [Script(Latin, code=Latn)]
dutch.script_codes # ["Latn"]
dutch.canonical_scripts # scripts marked canonical in LinguaMeta
# Geographic regions
dutch.regions # [Aruba, Belgium, ..., Netherlands, Suriname, ...]
dutch.country_codes # ["AW", "BE", "BQ", "CW", "NL", "SR", "SX"]
# Reverse traversal to script
latin = dutch.scripts[0]
latin.languoids # All languages using Latin script
# Cross-domain queries
dutch.languoids_with_same_script # other languages sharing any script
dutch.languoids_in_same_region # other languages in the same regions
Identifiers and Conversion
from qq import IdType
# Automatic detection
lang = db.guess("nld") # tries all identifier types
# Explicit conversion
db.convert("nl", IdType.BCP_47, IdType.ISO_639_3) # "nld"
db.convert("nld", IdType.ISO_639_3, IdType.GLOTTOCODE) # "dutc1256"
# Conversion where you don't know or care what the source is, just the target.
# Useful for normalizing multiple standards to one
db.convert("nl", IdType.ISO_639_3) # "nld"
db.convert("dutc1256", IdType.ISO_639_3) # "nld"
db.convert("mo", IdType.ISO_639_3) # "ron" (deprecated alias normalized silently)
# Full BCP-47-like tags and NLLB-style language-script tags are accepted by
# guess() and convert(); script/region subtags are ignored for languoid lookup.
db.guess("nl-Latn-NL").name # "Dutch"
db.convert("nl-Latn-NL", IdType.ISO_639_3) # "nld"
db.convert("nld_Latn", IdType.BCP_47) # "nl"
# If you want the language, script, and region components, use resolve_tag().
# Note that this only covers these three parts, not more specific region codes.
tag = db.resolve_tag("nl-Latn-NL")
tag.languoid.name # "Dutch"
tag.script.iso_15924 # "Latn"
tag.region.country_code # "NL"
# NLLB-style codes
dutch.nllb_codes() # ["nld_Latn"]
dutch.nllb_codes(use_bcp_47=True) # ["nl_Latn"]
get() and guess() warn when you use a deprecated code that still resolves to a replacement.
convert() does not; it silently normalizes deprecated aliases to the requested target identifier.
Multilingual Names
# Name of Dutch in French
dutch.name_in("fr") # "néerlandais"
dutch.name_in(french) # also accepts a Languoid object
# Native name
dutch.endonym # "Nederlands"
Command Line Interface
# Look up a language
qq get nl
qq get nld --type ISO_639_3
# Search by name or identifier
qq search Dutch
qq search nl
# Database statistics and validation
qq validate
# Rebuild the database from sources
qq rebuild
# Check source status
qq status
# Update sources (only needed if you want to rebuild the database,
# not necessary in normal use)
qq update
# For publishing the web-based explorer
qq export-demo
qq publish-demo <output-path>
qq prepare-release
Examples
See the examples/ directory for runnable scripts covering:
01_basic_usage.py: Loading and accessing attributes02_identifiers.py: Working with identifier types and retired codes03_conversion.py: Converting between identifiers04_traversal.py: Language family navigation05_search.py: Searching and filtering06_names.py: Multilingual name data07_geographic.py: Geographic regions and countries08_relations.py: Relationship graph traversal09_advanced_queries.py: Complex queries and statistics10_linking_datasets.py: Joining datasets that use different identifier systems11_normalizing_datasets.py: Normalizing mixed identifier codes to a single standard
Case studies
The browser explorer supports metadata inspection and graph traversal. Language pages link to datasets containing that language.
The case-studies/ directory contains runnable analyses that use qq:
huggingface-audit/: Scans 119,583 HuggingFace datasets with language tags and classifies 8,237 uniquelanguage:tags as valid, deprecated, a misused country code, or unknown. qq resolves 8,144 tags (98.9%) and normalizes coverage for 7,943 languoids.linking-datasets/: Links five lexical datasets (Concepticon, WordNet, Etymon, Phonotacticon, NoRaRe) that use different identifier standards. qq resolves all NoRaRe language codes and finds 34 languages covered by all five datasets.latex-tables/: Generates a LaTeX table of language metadata (identifiers, scripts, speaker counts, families) for an imaginary 30-language NLP benchmark.identifier-coverage/: Visualizes which combinations of identifier standards (Glottocode, ISO 639-3, ISO 639-1, Wikidata) cover which languoids as an UpSet plot.
Sources
This project builds on the work of many people. See docs/sources.md for the full source list, licenses, and terms.
Development
To rebuild the database from sources, install with the build extras:
uv add qwanqwa[build]
# or
pip install qwanqwa[build]
To install for local development:
git clone https://github.com/WPoelman/qwanqwa
cd qwanqwa
uv sync --group dev
License
The data sources qq incorporates have different licenses, see here.
We follow this example and license the software as Apache 2.0 and the data as CC BY-SA 4.0.
This means for instance that any data issues we encounter will be openly reported to the upstream sources (in accordance with ShareAlike principles of CC BY-SA), but that the software will ship with a compiled dataset (in accordance with the redistribution CC BY and CC BY-SA allow).
Ideally we'd use CC BY-SA for everything, but this is highly discouraged for software, even by Creative Commons themselves.
Metadata
Release files for qwanqwa 1.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| qwanqwa-1.3.0.tar.gz | 8.2 MB | Without provenance | Details for qwanqwa-1.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| qwanqwa-1.3.0-py3-none-any.whl | Python 3 | none | any | Details for qwanqwa-1.3.0-py3-none-any.whl |
Total release size: 16.4 MB
Release files / qwanqwa-1.3.0.tar.gz
| Download URL | qwanqwa-1.3.0.tar.gz |
|---|---|
| Size | 8.2 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f1332e93a7ffdd4ff18c2232054c26bdb19c1026cf0dead7efcb805020fc22ee
|
|
BLAKE2b-256 checksum How to use checksums |
fd249ba4b7f1d5ff4b5b2453b4244578167d7cbf22af80c6cd1599d04a5f552b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.9.27 {"installer":{"name":"uv","version":"0.9.27","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / qwanqwa-1.3.0-py3-none-any.whl
| Download URL | qwanqwa-1.3.0-py3-none-any.whl |
|---|---|
| Size | 8.2 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
854630adb5747f74d2e5d63b88e54d6cb52fc5aa3d1f0dfb60781e11b4da2536
|
|
BLAKE2b-256 checksum How to use checksums |
523c1ff59fe3fca8a465291b117eb248a1d8feafd34eb6840832bac046d2fe69
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.9.27 {"installer":{"name":"uv","version":"0.9.27","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for qwanqwa 1.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.