Install

To use this package, you minimally should have the latest version of Python 2.7 or any 3.5+ branch installed. The package is expected to work with both Python 2.7 and 3.5+, tested against those latest Python branches, as well as Python 3.5. The easiest way to get segtok installed is using pip or any other package manager that works with PyPI:

pip3 install segtok

Important: If you are on a Linux machine and have problems installing the regex dependency of segtok, make sure you have the python-dev and/or python3-dev packages installed to get the necessary headers to compile the package.

Then try the command line tools on some plain-text files (e.g., this README) to see if segtok meets your needs:

segmenter README.rst | tokenizer

Test Suite

The testing environment works with pytest, tox and pyenv. You first need to install pyenv (on OSX with Homebrew: brew install pyenv), and tox with pytest (pip3 install tox pytest). Configuring pyenv depends on the Python versions you have installed. Here, we assume you have the latest 2.7 and 3 versions installed and only need to provide an environment for testing segtok against the 3.8 branch:

pyenv install 3.8.2
pyenv global system 3.8.2

The second command is essential and indicates that your preferred Python binary is the system version and then the 3.8.2 branch. If you forget the second command, you will see errors like ERROR: InvocationError: Failed to get version_info for python3.8: pyenv: python3.8: command not found when running tox. If you only have one Python version installed (say, 2.7), to fully run the tests, you must also install and globally configure the other version (e.g., the latest 3.x) with pyenv, too.

Finally, to run all of segtok’s unit-test suite, just run tox:

tox

Usage

For details, please refer to the respective documentation; This README only provides an overview of the provided functionality.

Legal

License: MIT

Copyright (c) 2014-2021, Florian Leitner. All rights reserved.

Contributors (kudos):

• Mikhail Korobov (@kmike; port to Python2.7 and Travis CI integration)

• Georg Kucsko (@gkucsko; splitting sentences at terminals followed by noise)

• Karthikeyan Singaravelan (@tirkarthi; removing deprecation warnings, #23)

• Primož Godec (@PrimozGodec; fixed LICENSE file in setup.py)

History

• 1.5.11 setup.py: renamed data_files with the LICENSE.txt file reference to license_files

• 1.5.10 removed deprecation warning (#23) as well as support for Python 3.3 from tox

• 1.5.9 added the license as a LICENSE.txt file to this repository

• 1.5.7 enhancement: split sentences even if the terminal is followed by invalid characters (contributed by @gkucsko)

• 1.5.6 fixed a bug that would lead to joining lines in single-line mode (#11, reported by @yucongo)

• 1.5.5 support for middle name initials (“Lester P. Pearson”)

• 1.5.4 also support for European-style number-dates with numeric months (24. 12. 2016)

• 1.5.3 added support for European-style number-dates and for months (24. Dez. 2016)

• 1.5.2 fixed a tokenizer bug when parsing URLs ending with root paths (/), prevented sentence splitting after U.K., U.S. and E.U. if followed by upper-case (“U.S. Air Force”), added missing Unicode hyphens and apostrophes, and added test suite setup instructions

• 1.5.1 removed count_continuations.py discussion from README (was only confusing); the segmenter now can preserve tab-separated text IDs before the text itself when reading from STDIN and then inserts a (tab-separated) sentence ID column for each sentence printed to STDOUT: see segmenter option --with-ids

• 1.5.0 continuation words have been statistically evaluated and some poor choices removed (leading to more [precise] sentence splitting; see issue #9 by @Klim314 on GitHub)

• 1.4.0 the word_tokenizer no longer splits on colons between digits (time, references, …)

• 1.3.1 fixed multiple dangling commas and colons (reported by Jim Geovedi)

• 1.3.0 added Python2.7 support and Travis CI test integration (BIG thanks to Mikhail!)

• 1.2.2 made segtok.tokenizer.match protected (renamed to “_match”) and fixed UNIX linebreak normalization

• 1.2.1 the length of sentences inside brackets is now parametrized

• 1.2.0 wrote blog “documentation” and added chemical formula sub/super-script functionality

• 1.1.2 fixed Unicode list of valid sentence terminals (was missing U+2048)

• 1.1.1 fixed PyPI setup (missing MANIFEST.in for README.rst and “packages” in setup.py)

• 1.1.0 added possessive-s marker and apostrophe contraction splitting of tokens

• 1.0.0 initial release