Very compact Japanese tokenizer
See INSTALL file.
Example code for direct usage:
> import tinysegmenter > segmenter = tinysegmenter.TinySegmenter() > print ' | '.join(segmenter.tokenize(u"私の名前は中野です")) 私 | の | 名前 | は | 中野 | です
“TinySegmenter in Python”‘s interface is compatible with NLTK’s TokenizerI, although the distribution file below does not directly depend on NLTK. If you’d like to use it as a tokenizer in NLTK, you have to modify the first few lines of the code as below (so you can’t use the pypi repository version for now, if you wish to do this. Get the sources.):
import nltk import re from nltk.tokenize.api import * class TinySegmenter(TokenizerI):