Text tokenizer and sentence splitter

Library “text-sentence” is text tokenizer and sentence splitter.

Input is for main function is text, list of known names and abbreviations. Result is list of tokens. Each token has type and other attributes i.e.:

• is word,

• is number,

• is roman number,

• is sentence end,

• is abbreviation,

• is name,

• is contraction,

• is end of chapter

• etc.

Determining end of sentence needs special logic and care what is the main reason for naming package with “text-sentence”.

AUTHOR

Robert Lujo, Zagreb, Croatia, find mail address in LICENCE

FEATURES

To name the most important:

• TODO: …

System is based on unicode strings.

Check Getting started.

INSTALLATION

Installation instructions - if you have installed pip package http://pypi.python.org/pypi/pip:

pip install text-sentence

If not, then do it old-fashioned way:

• download zip from http://pypi.python.org/pypi/text-sentence/

• unzip

• open shell

• go to distribution directory

• python setup.py install

Development version you can see at http://bitbucket.org/trebor74hr/text-sentence.

or Mercurial clone with:

hg clone https://bitbucket.org/trebor74hr/text-sentence

GETTING STARTED

Usage example - start python shell:

>>> from text_sentence import Tokenizer
>>> t = Tokenizer()
>>> list(t.tokenize("This is first sentence. This is second one!And this is third, is it?"))
[T('this'/sent_start), T('is'), T('first'), T('sentence'), T('.'/sent_end),
 T('this'/sent_start), T('is'), T('second'), T('one'), T('!'/sent_end),
 T('and'/sent_start), T('this'), T('is'), T('third'), T(','/inner_sep),
 T('is'), T('it'), T('?'/sent_end)]

More samples can be found in tests:

http://bitbucket.org/trebor74hr/text-sentence/src/tip/text_sentence/test_sentence.txt

DOCUMENTATION

Currently there is no documentation. In progress …

SUPPORT

Since this project is limited by my free time, support is limited.

CONTRIBUTION

Since this project is not currently in the stable API phase, contribution should wait for a while.

RUNNING TESTS

All tests are doctests (not unittests). There are two type of tests in the package:

1. doctests in module i.e. in __init__.py

2. doctests in test_sentence.txt

Running module directly will run 1. and 2.

To run tests:

• goto text_sentence directory

• run tests by running module, e.g.:

  > python __init__.py
  __main__: running doctests
  test_sentence.txt: running doctests

• other with:

  > python -m"text_sentence"

TODO

various things, see readme.txt in dev version for details.

CHANGES