This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

“TinySegmenter in Python” is a Python port by Masato Hagiwara of TinySegmenter, which is an extremely compact Japanese tokenizer originally written in JavaScript by Mr. Taku Kudo.

The library has been finally packaged by Jehan. It resulted into this fork because Masako Hagiwara did not answer emails, and packaging patches could therefore not be committed upstream. But this is a friendly fork, and Masako Hagiwara is welcome to take back maintainance over his project. For the time being, I (Jehan) took up the maintainance, so please refer to this new website as being official, and direct any new patch there. I will follow up on patchs and bug reports, but probably won’t maintain an active development. Anyone wishing to improve the library is welcome to participate and will be gladly given committer rights.

It works on Python 2.5 or above (works on Python 3 too).

Authors

See all authors and contributors in AUTHORS file.

Download and Installation

This library can be installed the common ways: with a setup.py, as a pip package… See the INSTALL file in the package for more details.

If you simply want to download the source package, refer to the pypi repository: http://pypi.python.org/pypi/tinysegmenter

Development version can be downloaded anonymously at the Git repository:

$ git clone git://git.tuxfamily.org/gitroot/tinysegmente/tinysegmenter.git

or browsed online at: http://git.tuxfamily.org/tinysegmente/tinysegmenter/

Usage

Example code for direct usage:

> import tinysegmenter
> segmenter = tinysegmenter.TinySegmenter()
> print(' | '.join(segmenter.tokenize(u"私の名前は中野です")))
私 | の | 名前 | は | 中野 | です

TinySegmenter‘s interface is compatible with NLTK’s TokenizerI class, although the distribution does not directly depend on NLTK. Here is one way to use it as a tokenizer in NLTK (order of the multiple base classes matters):

import nltk.tokenize.api

class myTinySegmenter(tinysegmenter.TinySegmenter, nltk.tokenize.api.TokenizerI):
    pass
segmenter = myTinySegmenter()
# This segmenter can be used any place which expects a NLTK's TokenizerI subclass.

For more about NLTK (Natural Language Toolkit module), see: http://nltk.org/api/nltk.tokenize.html#nltk.tokenize.api.TokenizerI

Contact, Bugs and Contributing

All bug, patch, question, etc. can be sent to tinysegmenter at zemarmot dot net.

License

This package is distributed under a New BSD License (see COPYING file).

Release History

Release History

0.2

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
tinysegmenter-0.2.tar.gz (15.7 kB) Copy SHA256 Checksum SHA256 Source Oct 11, 2014

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting