cppjieba-mmap

Linux native Chinese tokenizer for CPython 3.13 and later. Version 0.1.0 bundles a merged Taiwan dictionary and uses CppJieba Mix segmentation with HMM. The dictionary DAT uses a read-only shared memory mapping. Workers reuse its file-backed pages instead of allocating a private dictionary trie per process. HMM tables remain process-private.

Release wheels target manylinux_2_28 x86_64 and aarch64 for CPython 3.13, 3.14 and free-threaded 3.14. Wheels need no compiler at installation time. Source builds use setuptools, pybind11 3.0.1 and a C++17 compiler, without CMake. Runtime needs no OpenCC, jieba, network dictionary downloads or model server.

Install from source

Run these commands from this repository root:

uv venv --python 3.14
uv pip install '.[test]'
.venv/bin/python -m pytest

import cppjieba_mmap

tokenizer = cppjieba_mmap.Tokenizer(
    str(cppjieba_mmap.DICT_PATH),
    str(cppjieba_mmap.HMM_PATH),
    "/tmp/my-private-tokenizer-cache",
)
words = tokenizer.cut("超商取貨")

DICT_PATH and HMM_PATH are readable bundled pathlib.Path objects from import time. DICT_SHA256 identifies the exact uncompressed dictionary bytes. The dictionary is packaged uncompressed. No runtime conversion occurs. Its 582704 entries combine the pinned CppJieba dictionary with an OpenCC s2twp conversion of the supplied jieba-next dictionary. Duplicate surfaces retain the larger frequency. See PROVENANCE.md for exact inputs and hashes.

Output contract

cut(text: str) -> list[str] uses Jieba::Cut(text, words, true) (Mix + HMM), then removes whitespace-only tokens, matching the cpp-tw retrieval evaluation. Other token text and ordering stay unchanged. English case is preserved. Empty and whitespace-only inputs return []. In particular, 256GB remains 256 / GB; the wrapper does not silently alter this evaluated behavior.

Cache and concurrency

Cache names contain dictionary SHA-256, the native cache format revision, architecture and endianness. Changed dictionary bytes cannot select the same DAT. HMM data is loaded separately and does not determine the DAT identity.

Every constructor locks a per-key flock file. Exactly one simultaneous constructor builds a missing DAT. A private temporary directory holds the dictionary snapshot and incomplete DAT. Publication uses an atomic rename within the cache filesystem after validation, fsync and chmod 0444. Readers attach using PROT_READ | MAP_SHARED. Locks remain on disk for reuse. Truncated or header-mismatched cache files are rebuilt under the same lock. Existing instances retain their mapped inode while a replacement is published.

Use a trusted local filesystem and one OS user for all workers. Cache directories must not be writable by other users. Input models must be trusted and remain unchanged while constructing a tokenizer. Dictionary bytes are snapshotted even on cache hits, avoiding hash/read races. Temporary snapshots are deleted after construction. A killed process can leave a private .build-* directory; it is never treated as a published cache. Remove such directories only while no constructors run. The package does not evict older DAT files.

Threads and free threading

The binding releases the GIL during native construction and cutting. One instance supports concurrent cut calls: the engine exposes no mutators, dictionary/model lookups are const, and cut/DAG/Viterbi buffers are call-local. The native module declares py::mod_gil_not_used(). The process-global upstream umask mutation is removed. Constructors use separate engine instances and per-cache file locks. Python result filtering runs after the native call returns.

This is not a claim of subinterpreter support or lock-free Python execution. Tests exercise two threads with a released GIL and three spawned processes. HMM tables and per-call buffers remain process-private; only the DAT is shared.

Limits

The DAT header is validated, but this is not a sandbox for malicious native cache contents or untrusted model files. Do not reuse caches written by other users. Inputs consume memory proportional to their length. No Windows support, user-dictionary mutation, POS tagging API, query/full mode or cache eviction is exposed. See PROVENANCE.md for source pins, patches and third-party licenses.

Releases

CI builds and tests one CPython 3.14 x86_64 wheel on pushes and pull requests. The release workflow tests all six Python/architecture wheels on native Linux runners, builds an sdist, and publishes only after every build succeeds.

Before the first release, configure a PyPI Trusted Publisher for project cppjieba-mmap, owner coseto6125, repository cppjieba-mmap, workflow release.yml, and environment pypi. Configure that GitHub environment with release approval protection. No PyPI API token is required.

Set the package version in pyproject.toml, then push the matching v<version> tag. For this version the tag is v0.1.0. The workflow rejects a version mismatch. Only the isolated publish job receives id-token: write.

License

Package glue is MIT-licensed under LICENSE. Bundled source, dictionary and conversion resources retain their own notices in licenses/ and PROVENANCE.md. The distribution's combined license expression includes these third-party licenses; it does not relicense them as MIT.