thaiphon-data-volubilis 0.2.0

Exact-match Thai lexicon derived from the VOLUBILIS Mundo Multilingual Thai Dictionary, for use with thaiphon.

thaiphon-data-volubilis

An exact-match Thai pronunciation lexicon for the thaiphon phonological engine, derived from the VOLUBILIS Mundo Multilingual Thai Dictionary & Database by Belisan. When installed alongside thaiphon, the pipeline short-circuits on words it finds in this data and returns the exact reading instead of running the rule-based derivation.

Installing it lifts thaiphon's Wiktionary-IPA exact-match rate from ~57 % on the base engine to ~75 %. That gain is the word-boundary and variant coverage a rule-based engine can't infer from orthography alone.

Install

pip install thaiphon-data-volubilis
# or
uv add thaiphon-data-volubilis

Python 3.10+, no runtime dependencies. thaiphon picks the lexicon up automatically on import; no configuration needed.

What's in the wheel

A single lexicon.db file: a read-only SQLite database of 84 k Thai words, each mapped to its pre-derived phonological word (syllable-segmented, with onsets, vowels, codas, and tones resolved). Keys are Thai strings; the primary-key index over thai_word gives O(log n) lookup in tens of microseconds.

The package exposes one Mapping[str, PhonologicalWord] named ENTRIES with the usual dict-like API (__getitem__, __contains__, get, keys, items, values, __len__, __iter__). Callers don't see the storage backend, so a future release can change it without breaking anyone.

Memory footprint

The lexicon is memory-mapped via SQLite's mmap_size pragma and opened with immutable=1. Every process that imports the package shares the same underlying pages through the operating-system page cache. Numbers from this machine, against the previous in-memory representation:

Measurement	Before	After
Per-process RSS after first lookup	~340 MiB	~7 MiB
Per-process RSS after 1 k lookups	~340 MiB	~35 MiB
Wheel size	1.6 MiB	2.7 MiB
.pyc on Python 3.10 (import cost)	117 MiB	0

The ~35 MiB after warmup is the per-process LRU of the last 10 k inflated entries. The ~50 MiB or so of physical memory holding the mmap'd database pages is counted once by the kernel and shared by every Python worker that has imported the package.

What this means in practice

For multi-worker web servers (gunicorn, uvicorn with workers, uWSGI) total lexicon RSS is bounded by the size of lexicon.db plus a small per-worker LRU. It does not multiply with worker count. Dozens of workers on a single host stay practical.

SQLite forbids sharing a connection across threads, so each Python thread that touches the lexicon lazily opens its own connection via threading.local(). FastAPI / Starlette's threadpool, Django's async views, and any other thread model work without extra plumbing.

Serverless cold starts (AWS Lambda, Cloud Run, Fargate, and Cloudflare Workers when Python support allows) don't pay a .pyc unpack or a lexicon-inflation cost. The file is only read from as entries are requested.

Pre-forking servers like sync gunicorn and uWSGI inherit no SQLite handle across fork(), since the connection lives in threading.local. Each child opens its own on first use.

Attribution

Source data: VOLUBILIS Mundo Multilingual Thai Dictionary & Database by Belisan — https://belisan-volubilis.blogspot.com/.

License

The data is distributed under CC BY-SA 4.0, matching the upstream source license. Derivative works of this data must also be licensed under CC BY-SA 4.0 or a compatible license. See LICENSE and NOTICE for details.