tokmor 1.4.3

Dependency-free deterministic tokenizer and morphology hints for 300+ configured languages

TokMor

Dependency-free deterministic tokenizer and morphology hints for multilingual text.

TokMor is a compact preprocessing library for deterministic tokenization, offsets, language-aware segmentation, and rule-based morphology hints. It is built for offline product use: deterministic output, no model downloads, and no runtime dependencies in the core package.

TokMor is not a Transformer model, POS tagger, NER classifier, crawler, dashboard, or news-analysis pipeline. Downstream systems can use TokMor spans and morphology hints before entity recognition, search, RAG, or analytics.

Current Scope

• 300+ configured language codes with deterministic fallback tokenization.
• 121-language product contract gate for representative real-world text patterns.
• Repository regression suites for morphology hints, boundaries, offsets, and multilingual product-contract cases.
• Focused no-space language support for Korean, Japanese, Chinese, Thai, and related CJK/SEA scripts.
• Stable handling for offsets, punctuation, dates, URLs, versions, units, abbreviations, and common clitic/MWT patterns.
• CLI, Python API, optional FastAPI server, and stdio MCP server.

Language quality is tiered. Some languages have tuned segmentation and morphology packs; others intentionally use conservative deterministic fallback rules. See docs/LANGUAGE_TIERS.md and docs/COMMERCIAL_MVP.md.

Install

pip install tokmor

The core wheel includes compact starter dictionaries for CJK/SEA segmentation, frequent-word POS/UPOS hints, MWT/clitic splits, lemma manifests, and small domain hint assets. No separate dictionary download is required for default local use. Larger lemma dictionaries and language resources remain optional offline data packs.

Run tokmor assets --pretty after installation to inspect the bundled dictionary inventory.

From a source checkout, install editable mode from the repository root:

python3 -m pip install -e ".[dev]"

Python

import tokmor

out = tokmor.unified_tokenize("우리는 2025-01-10에 서울을 방문했다.", lang="ko", sns=False)
print([t["text"] for t in out["tokens"]])

prep = tokmor.ner_preprocess("東京の新製品を25℃でテストした。", lang="ja")
print(prep[:3])

Token dictionaries include text, start, and end; offsets slice back to the original string. ner_preprocess prepares spans for downstream NER/search systems; it does not assign entity labels.

CLI

tokmor tokenize --lang ko --text "로마로부터 정명석의" --text-output
tokmor analyze --lang ja --text "東京の新製品" --pretty
tokmor ner-preprocess --lang zh --text "我们在2025-01-10访问了首尔。" --pretty
tokmor explain --lang auto --text "TokMor checks bundled assets." --pretty
tokmor explain-document --lang auto --text "TokMor checks bundled assets. It keeps offsets stable." --summary-only --pretty
tokmor assets --pretty

If the console script is not on PATH, use the equivalent module form: python -m tokmor tokenize ....

See docs/CLI.md.

MCP and API

Run the MCP server for AI hosts:

tokmor-mcp

Run the optional HTTP API after installing the api extra:

python3 -m pip install "tokmor[api]"
tokmor-api --host 127.0.0.1 --port 8000

Contract details are in docs/MCP.md and docs/TOKMOR_API_CONTRACT.md.

Verify a Checkout

Fast product checks:

PYTHONPATH=TokMor_v1 python3 examples/quickstart_tier1.py
PYTHONPATH=TokMor_v1 python3 scripts/run_token_invariants_smoke.py
PYTHONPATH=TokMor_v1 python3 scripts/run_multilang_product_contract.py
PYTHONPATH=TokMor_v1 python3 scripts/run_morph_regress.py --langs en,ko,ja,zh,th,ar,hi
PYTHONPATH=TokMor_v1 python3 scripts/audit_repo_scope.py --strict --include-history

Full release gate:

python3 scripts/run_productization_checks.py \
  --python python3 \
  --with-pos-eval \
  --pos-ud-root /path/to/ud_top10 \
  --with-bench \
  --with-package \
  --bench-iters 300

Repository Hygiene

This repository is kept to TokMor core runtime, product contracts, regression fixtures, packaging, and product documentation. Non-core dashboards, news-analysis apps, generated build outputs, local data, and session artifacts are intentionally excluded. Run the scope audit before release:

PYTHONPATH=TokMor_v1 python3 scripts/audit_repo_scope.py --strict --include-history --json

The expected Git author/committer identity for this repository is Zeus Kim <[e-mail]>.

Docs

• docs/COMMERCIAL_MVP.md: conservative product positioning and limits.
• docs/AUTODEV_AUTOMATION.md: automated d2 development loop.
• docs/NEXT_DEVELOPMENT_PLAN.md: parallel development tracks and gates.
• docs/LANGUAGE_TIERS.md: language support model.
• docs/TOKMOR_API_CONTRACT.md: Python/API/MCP contract expectations.
• docs/AUTONER_TOKMOR_HANDOFF.md: downstream AutoNER integration notes.
• docs/DATA_SOURCES_AND_LICENSES.md: data source and license notes.
• docs/FAQ.md: what TokMor is and is not.

License

MIT. See LICENSE.