cloud-language-tools-wenlin

Wenlin Chinese-English dictionary data and parser, packaged for Cloud Language Tools.

The dictionary database (202,256 entries, 235,594 definitions) is bundled inside the wheel. There is no download step at install time and no external data directory to provision.

pip install cloud-language-tools-wenlin

The import name is clt_wenlin.

Usage

Look up a word by its simplified or traditional form:

import json
import clt_wenlin

connection = clt_wenlin.open_connection()
for (entry,) in connection.execute(
        'SELECT entry FROM words WHERE simplified=?', ('你好',)):
    print(json.loads(entry))
connection.close()

{'simplified': '你好',
 'traditional': '你好',
 'pinyin': 'nị̌',
 'parts_of_speech': [{'part_of_speech': 'intj.',
                      'definitions': [{'definition': 'How are you?; Hello.'}]}]}

Search the English definitions, which are indexed with sqlite FTS5:

connection.execute(
    "SELECT definition, entry_id FROM definitions WHERE definitions MATCH 'confirmation'")

If you would rather manage the connection yourself, clt_wenlin.get_wenlin_db_path() returns the filesystem path of the bundled database.

Database schema

Each entry document looks like:

{
    'simplified': str,
    'traditional': str,
    'pinyin': str,
    'parts_of_speech': [{
        'part_of_speech': str | None,
        'definitions': [{
            'definition': str,
            'example_pinyin': str,       # optional
            'example_chinese': str,      # optional
            'example_translation': str,  # optional
            'measure_word': str,         # optional
        }],
    }],
}

Parsing a cidian.u8 file

The parser that produced the database is part of the package:

import clt_wenlin

entries = clt_wenlin.read_dictionary_file('cidian.u8')
clt_wenlin.create_sqlite_file('cidian.u8', 'wenlin.db')

Only the English side of the dictionary is kept; French definitions are dropped.

Development

The project uses uv:

uv sync
uv run pytest

The database is committed to the repository xz-compressed as data/wenlin_revA.db.xz (16MB). The build hook in hatch_build.py decompresses it into src/clt_wenlin/data/ so that it lands, uncompressed and ready to query, in the wheel. To rebuild it from a Wenlin source file:

uv run python tools/build_db.py /path/to/cidian.u8

Bump WENLIN_DB_REV in src/clt_wenlin/database.py when the data changes.

Releasing

./package.sh 2.0.1

License

GPL-3.0-or-later. The Wenlin dictionary data is the property of Wenlin Institute.