IDSpiece

漢字/汉字-Tokenizer with Ideographic Description Sequence from CHISE-IDS.

Only nine IDCs (U+2FF0, U+2FF1, U+2FF4 to U+2FFA) are used.
IDCs never occur instantly after another IDC.
Instantly after IDCs, Kanxi Radicals and Supplement (U+2E80 to U+2FD5) are preferred.
Otherwise, CJK Unified Ideographs and Extension A (U+3400 to U+9FFC) are preferred.

Basic usage

>>> from idspiece import idstable
>>> def tokenize(text):
...   tokens=[]
...   while text>"":
...     c=text[0]
...     if c in idstable:
...       tokens.append(idstable[c][0:2])
...       text=idstable[c][2]+text[1:]
...     else:
...       tokens.append(c)
...       text=text[1:]
...   return tokens
...
>>> t=tokenize("羯諦羯諦波羅羯諦波羅僧羯諦菩提薩婆訶")
>>> print(t)
['⿰⽺', '⿱⽈', '⿹⼓', '亾', '⿰⾔', '帝', '⿰⽺', '⿱⽈', '⿹⼓', '亾', '⿰⾔', '帝', '⿰⺡', '皮', '⿱⺲', '⿰⽷', '隹', '⿰⽺', '⿱⽈', '⿹⼓', '亾', '⿰⾔', '帝', '⿰⺡', '皮', '⿱⺲', '⿰⽷', '隹', '⿰⺅', '曾', '⿰⽺', '⿱⽈', '⿹⼓', '亾', '⿰⾔', '帝', '⿱⺾', '⿱⽴', '口', '⿰⺘', '⿱⽇', '⿱⼀', '龰', '⿱⺾', '⿰⻖', '⿸产', '生', '⿱波', '女', '⿰⾔', '可']

Installation

pip3 install idspiece

Author

Koichi Yasuoka (安岡孝一)

Release files for idspiece 0.6.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for idspiece 0.6.5
File	Interpreter	ABI	Platform	Reset
idspiece-0.6.5-py3-none-any.whl 932.7 kB Feb 4, 2022	Python 3	none	any	Details

Release files / idspiece-0.6.5-py3-none-any.whl

Download URL	idspiece-0.6.5-py3-none-any.whl
Size	932.7 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`4aec710919dc9b5aebee6f93402f00a9342191ba63c53308cd3e01b1ff0f5750`
BLAKE2b-256 checksum How to use checksums	`10f7137cde082f5789de0251ab24910b5acdfa367facaf99844bbb6657d8630c`
Upload date	Feb 4, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/52.0.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.2`