c++ mosestokenizer
Project description
opus-fast-mosestokenizer is a fork of fast-mosestokenizer created to ensure compability of the package with current Python environments.
fast-mosestokenizer is a C++ implementation of the moses tokenizer which is a favourite among the folks in NLP research.
The reason for using this package over the original perl implementation is for the purpose of portability. With the C++ source code, you can use this library basically in every language.
The C++ script was adapted from the mosesdecoder repository
contrib/c++tokenizer
.
Benchmark
fast-mosestokenizer is also fast.
On english, it is about 6x faster than tokenizer.perl
and 15x faster than
sacremoses
.
see ./bench/README.md for more information.
Installation
Python users using linux
and osx>=10.15
can install directly from PyPI.
pip install fast-mosestokenizer
See ./INSTALL.md for more information.
Usage (Command-line tool)
# Piping is the standard way to configure input and output stream.
# mosestokenizer would apply tokenization to each line of the input stream.
mosestokenizer en < infile > outfile
# For a full list of options, refer to the help message.
mosestokenizer -h
Usage (Python)
# Usage patterns are mostly the same as sacremoses.
>>> from mosestokenizer import MosesTokenizer
>>> tokenizer = MosesTokenizer('en')
>>> tokenizer.tokenize("""
The English name of Singapore is an anglicisation of the native Malay name for
the country, Singapura, which was in turn derived from the Sanskrit word for
lion city (romanised: Siṃhapura; Brahmi: 𑀲𑀺𑀁𑀳𑀧𑀼𑀭; literally "lion city"; siṃha
means "lion", pura means "city" or "fortress").[8]
""")
[
'The', 'English', 'name', 'of', 'Singapore', 'is', 'an', 'anglicisation',
'of', 'the', 'native', 'Malay', 'name', 'for', 'the', 'country', ',',
'Singapura', ',', 'which', 'was', 'in', 'turn', 'derived', 'from', 'the',
'Sanskrit', 'word', 'for', 'lion', 'city', '(', 'romanised', ':',
'Siṃhapura', ';', 'Brahmi', ':', '𑀲𑀺𑀁𑀳𑀧𑀼𑀭', ';', 'literally', '"', 'lion',
'city', '"', ';', 'siṃha', 'means', '"', 'lion', '"', ',', 'pura', 'means',
'"', 'city', '"', 'or', '"', 'fortress', '"', ')', '.', '[', '8', ']'
]
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for opus_fast_mosestokenizer-0.0.8.6.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3fb5a577c34c55521a7eb3859085508490f7adfd2a59db1b649137088d7fdc0f |
|
MD5 | 42b187db4ee62c8cafcf51f48ae4bfa7 |
|
BLAKE2b-256 | 8e89b171790dd876251f3621cc175afc07b7a4c217e0a90e92308bc056f0bd2c |
Hashes for opus_fast_mosestokenizer-0.0.8.6-cp312-cp312-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 205668b43f14d5b2cffd9c4484ad7e2e8fea9f351968d2a27ae6b0c84341b87d |
|
MD5 | 5d3f16abadd7c0bb3d1b475bea295b15 |
|
BLAKE2b-256 | b8d994c7dcd93569c7b4a1c543f0a812af91fad3ff6d34214b43146811a87589 |
Hashes for opus_fast_mosestokenizer-0.0.8.6-cp312-cp312-macosx_12_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2060f74cc2ef2ee9e23eb346d431cdfbcba9bd6fd4e1dd82a35adbac717067f6 |
|
MD5 | 3c30ce0f178edc75c8c424ab61e8b61b |
|
BLAKE2b-256 | 62eeb23b703bd5830a981aa1479135b9b1f2ce6e870ef1bf4c21a45d05571372 |
Hashes for opus_fast_mosestokenizer-0.0.8.6-cp311-cp311-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 14641a228dd16bd9646c029fc351ba1d0a677b6e75db7f01b7b6dbab7050c85f |
|
MD5 | f2aa46317a9d03f8e40097ae855e3fb1 |
|
BLAKE2b-256 | 3d090443e63f848191adb0c31efcb7e7168add6669a2c9e70671da1e89fc43a6 |
Hashes for opus_fast_mosestokenizer-0.0.8.6-cp311-cp311-macosx_12_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4e288d89471991425183786476e05d62a0150db9e7b55fad0b1a14d42f95bf19 |
|
MD5 | 7294c895a01ef3e075cdafc09131ed17 |
|
BLAKE2b-256 | a336482af33f75a67ea5707ef4b6e3b10e7c743a290bbe126a4dcf18763b1c21 |
Hashes for opus_fast_mosestokenizer-0.0.8.6-cp310-cp310-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 12892b9540ac5ab45fb3dfc6e19ee6bec7bb1f8b9bf5d18e82f55dc20a31cb0f |
|
MD5 | 4fbc808364c1af34d5b352bbe86cb136 |
|
BLAKE2b-256 | 5867f9dfbb91321fefb13e85ae4679d316c13bbf1a2a7e9ec630e33b0f640e59 |
Hashes for opus_fast_mosestokenizer-0.0.8.6-cp310-cp310-macosx_12_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a57b3744c92f1a03245f347eb98a967861d818366d4564716639ef0d171dc8a3 |
|
MD5 | 7140d935c4216beb811e1185a838b75d |
|
BLAKE2b-256 | e53c7582487fa77045d001152f7674343bdb1ef0e9efa189ba1a27cb65e5e8db |
Hashes for opus_fast_mosestokenizer-0.0.8.6-cp39-cp39-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2067e3574ceff82cace4b066187d0797dde909756c2f705ebe82b07c3b3277d9 |
|
MD5 | b26a84dab716045b3968325b5de72c8a |
|
BLAKE2b-256 | c82371e956af61b895984efce6f0695351869dd1cfd944b2393ee7ca27c01f5a |
Hashes for opus_fast_mosestokenizer-0.0.8.6-cp39-cp39-macosx_12_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 09c37ea8b294d4067907b37bab7181e2c65aaa1bc1a126521c79a17627a5d3b1 |
|
MD5 | cac3c3295a18ec2fb7568cdb896d36a8 |
|
BLAKE2b-256 | e152ed8be1f9475dc0a493f6f6f90df278c9daf15e0859cee5edd238f6ee5ccb |
Hashes for opus_fast_mosestokenizer-0.0.8.6-cp38-cp38-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6c9f673b5f66fee10b12923ba8c4324648082584f65e2815ad59b972a544dda5 |
|
MD5 | e09ab89a8e42dc17fd1740bf743f2ebe |
|
BLAKE2b-256 | 6c311a295e2451573fd7bbe5b19ef80b4ceeab810e921564ffdfbf608ad7eeab |
Hashes for opus_fast_mosestokenizer-0.0.8.6-cp38-cp38-macosx_12_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cac047a6997226b9452046787a1c0bc8005a5b731fd1bdf3ef1affa0ef2a0759 |
|
MD5 | 57e6035ce7106c7dca3fe3f6ed6dca24 |
|
BLAKE2b-256 | 6d3d5a2cc2b176e89f6bb934f24807a3343b79b9ace52cf1d60e987866eea333 |
Hashes for opus_fast_mosestokenizer-0.0.8.6-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 74fa9048d1e328cc072b30411da72931dbd171c367175c4b843ae6299b2ed0ac |
|
MD5 | 2a0fa266f20bf1990828aeab9e45c6a5 |
|
BLAKE2b-256 | 4cdc25f84105f325f6ef089b8adbcea670326df95d4925be9ff0d7956b3c3be2 |
Hashes for opus_fast_mosestokenizer-0.0.8.6-cp37-cp37m-macosx_12_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7002438571f3ede159ef9743bc404b5f366022804b657587a1bc845cc8a42714 |
|
MD5 | 8cfe728df607e7134195469cbaf1d9db |
|
BLAKE2b-256 | 63ade4751d90b17f31e02ee089c274750570eda7cddc9d1a8afa1ce8a6b0adaf |
Hashes for opus_fast_mosestokenizer-0.0.8.6-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 088cac2463724cff41adcaf4d0c4a86f67974299810e7210a154664ce1335890 |
|
MD5 | 34af88a496903bc6987acd946fb49405 |
|
BLAKE2b-256 | 9e7335bff274120ca7c3ec2eb6d332a8670bc6eeaada36aaa7f1a182ab31f5e7 |
Hashes for opus_fast_mosestokenizer-0.0.8.6-cp36-cp36m-macosx_12_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | be080dd4af290534fd032a6e50e0ce4f97007ae787356f5d85ec3cfed2de029b |
|
MD5 | 1fbd6db5b8a8d2db8b70a2c4b4f306f6 |
|
BLAKE2b-256 | ba8a35b0758c6b04d3ac0228565b5352864958cf492f2ea0269406b9a3f1a60b |