Skip to main content

Very compact Japanese tokenizer

Project description

“TinySegmenter in Python” is a Python port of TinySegmenter (which is an extremely compact (23KB) Japanese tokenizer originally written in JavaScript by Mr. Taku Kudo. It works on Python 2.5 or above.


See the original authors in AUTHORS file.


See INSTALL file.


Example code for direct usage:

> import tinysegmenter
> segmenter = tinysegmenter.TinySegmenter()
> print ' | '.join(segmenter.tokenize(u"私の名前は中野です"))
私 | の | 名前 | は | 中野 | です

“TinySegmenter in Python”‘s interface is compatible with NLTK’s TokenizerI, although the distribution file below does not directly depend on NLTK. If you’d like to use it as a tokenizer in NLTK, you have to modify the first few lines of the code as below (so you can’t use the pypi repository version for now, if you wish to do this. Get the sources.):

import nltk
import re
from nltk.tokenize.api import *

class TinySegmenter(TokenizerI):

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
tinysegmenter-0.1.tar.gz (12.2 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page