This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (
Help us improve Python packaging - Donate today!

Wrappers for several pre-processing scripts from the Moses toolkit.

Project Description

This package provides wrappers for some pre-processing Perl scripts from the Moses toolkit, namely, normalize-punctuation.perl, tokenizer.perl, detokenizer.perl and split-sentences.perl.

Sample Usage

All provided classes are importable from the package mosestokenizer.

>>> from mosestokenizer import *

All classes have a constructor that takes a two-letter language code as argument ('en', 'fr', 'de', etc) and the resulting objects are callable.

When created, these wrapper objects launch the corresponding Perl script as a background process. When the objects are no longer needed, you should call the .close() method to close the background process and free system resources.

The objects also support the context manager interface. Thus, if used within a with block, the .close() method is invoked automatically when the block exits.

The following two usages of MosesTokenizer are equivalent:

>>> # here we will call .close() explicitly at the end:
>>> tokenize = MosesTokenizer('en')
>>> tokenize('Hello World!')
['Hello', 'World', '!']
>>> tokenize.close()
>>> # here we take advantage of the context manager interface:
>>> with MosesTokenizer('en') as tokenize:
>>>     tokenize('Hello World!')
['Hello', 'World', '!']

As shown above, MosesTokenizer callable objects take a string and return a list of tokens (strings).

By contrast, MosesDetokenizer takes a list of tokens and returns a string:

>>> with MosesDetokenizer('en') as detokenize:
>>>     detokenize(['Hello', 'World', '!'])
'Hello World!'

MosesSentenceSplitter does more than the name says. Besides splitting sentences, it will also unwrap text, i.e. it will try to guess if a sentence continues in the next line or not. It takes a list of lines (strings) and returns a list of sentences (strings):

>>> with MosesSentenceSplitter('en') as splitsents:
>>>     splitsents([
...         'Mr. Smith is away.  Do you want to',
...         'leave a message?'
...     ])
['Mr. Smith is away.', 'Do you want to leave a message?']

MosesPunctuationNormalizer objects take a string as argument and return a string:

>>> with MosesPunctuationNormalizer('en') as normalize:
>>>     normalize('«Hello World» — she said…')
'"Hello World" - she said...'


Copyright ® 2016-2017, Luís Gomes <>.

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA

Release History

Release History

This version
History Node


History Node


History Node


Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
mosestokenizer-1.0.0-py3-none-any.whl (51.2 kB) Copy SHA256 Checksum SHA256 py3 Wheel May 24, 2017
mosestokenizer-1.0.0.tar.gz (33.7 kB) Copy SHA256 Checksum SHA256 Source May 24, 2017

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting