Skip to main content

Wrappers for several pre-processing scripts from the Moses toolkit.

Project description

This package provides wrappers for some pre-processing Perl scripts from the Moses toolkit, namely, normalize-punctuation.perl, tokenizer.perl, detokenizer.perl and split-sentences.perl.

Sample Usage

All provided classes are importable from the package mosestokenizer.

>>> from mosestokenizer import *

All classes have a constructor that takes a two-letter language code as argument ('en', 'fr', 'de', etc) and the resulting objects are callable.

When created, these wrapper objects launch the corresponding Perl script as a background process. When the objects are no longer needed, you should call the .close() method to close the background process and free system resources.

The objects also support the context manager interface. Thus, if used within a with block, the .close() method is invoked automatically when the block exits.

The following two usages of MosesTokenizer are equivalent:

>>> # here we will call .close() explicitly at the end:
>>> tokenize = MosesTokenizer('en')
>>> tokenize('Hello World!')
['Hello', 'World', '!']
>>> tokenize.close()
>>> # here we take advantage of the context manager interface:
>>> with MosesTokenizer('en') as tokenize:
>>>     tokenize('Hello World!')
...
['Hello', 'World', '!']

As shown above, MosesTokenizer callable objects take a string and return a list of tokens (strings).

By contrast, MosesDetokenizer takes a list of tokens and returns a string:

>>> with MosesDetokenizer('en') as detokenize:
>>>     detokenize(['Hello', 'World', '!'])
...
'Hello World!'

MosesSentenceSplitter does more than the name says. Besides splitting sentences, it will also unwrap text, i.e. it will try to guess if a sentence continues in the next line or not. It takes a list of lines (strings) and returns a list of sentences (strings):

>>> with MosesSentenceSplitter('en') as splitsents:
>>>     splitsents([
...         'Mr. Smith is away.  Do you want to',
...         'leave a message?'
...     ])
...
['Mr. Smith is away.', 'Do you want to leave a message?']

MosesPunctuationNormalizer objects take a string as argument and return a string:

>>> with MosesPunctuationNormalizer('en') as normalize:
>>>     normalize('«Hello World» — she said…')
...
'"Hello World" - she said...'

License

Copyright ® 2016-2021, Luís Gomes <luismsgomes@gmail.com>.

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mosestokenizer-1.2.1.tar.gz (37.1 kB view details)

Uploaded Source

File details

Details for the file mosestokenizer-1.2.1.tar.gz.

File metadata

  • Download URL: mosestokenizer-1.2.1.tar.gz
  • Upload date:
  • Size: 37.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/40.8.0 requests-toolbelt/0.8.0 tqdm/4.45.0 CPython/3.7.3

File hashes

Hashes for mosestokenizer-1.2.1.tar.gz
Algorithm Hash digest
SHA256 438b3e35a221f7930c408e97e3f38af6d0cec74b991eb9edb00a44e3510e836d
MD5 0004d7cb0200633ac0ce49d25683007a
BLAKE2b-256 8d844f3c1b5b8d796a07e3816cd41f7b1491e2291db4ade5f17b850116fd80e5

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page