tinysegmenter 0.4

Authors

See all authors and contributors in AUTHORS file.

Download and Installation

This library can be installed the common ways: with a setup.py, as a pip package… See the INSTALL file in the package for more details.

If you simply want to download the source package, refer to the pypi repository: http://pypi.python.org/pypi/tinysegmenter

Development version can be downloaded anonymously at the Git repository:

$ git clone git://git.tuxfamily.org/gitroot/tinysegmente/tinysegmenter.git

or browsed online at: http://git.tuxfamily.org/tinysegmente/tinysegmenter/

Usage

Example code for direct usage:

> import tinysegmenter
> segmenter = tinysegmenter.TinySegmenter()
> print(' | '.join(segmenter.tokenize(u"私の名前は中野です")))
私 | の | 名前 | は | 中野 | です

TinySegmenter‘s interface is compatible with NLTK’s TokenizerI class, although the distribution does not directly depend on NLTK. Here is one way to use it as a tokenizer in NLTK (order of the multiple base classes matters):

import nltk.tokenize.api

class myTinySegmenter(tinysegmenter.TinySegmenter, nltk.tokenize.api.TokenizerI):
    pass
segmenter = myTinySegmenter()
# This segmenter can be used any place which expects a NLTK's TokenizerI subclass.

For more about NLTK (Natural Language Toolkit module), see: http://nltk.org/api/nltk.tokenize.html#nltk.tokenize.api.TokenizerI

Contact, Bugs and Contributing

All bug, patch, question, etc. can be sent to tinysegmenter at zemarmot dot net.

License

This package is distributed under a New BSD License (see COPYING file).