Skip to main content

Rule-based sentence tokenizer for Russian language

Project description


A simple and fast rule-based sentence segmentation. Tested on OpenCorpora and SynTagRus datasets.


pip install rusenttokenize


>>> from rusenttokenize import ru_sent_tokenize
>>> ru_sent_tokenize('Эта шоколадка за 400р. ничего из себя не представляла. Артём решил больше не ходить в этот магазин')
['Эта шоколадка за 400р. ничего из себя не представляла.', 'Артём решил больше не ходить в этот магазин']


The tokenizer has been tested on OpenCorpora and SynTagRus. There are two important metrics.

Precision. First one is we took single sentences from the datasets and measured how many times tokenizer didn't split them.

Recall. Second metric is we took two consecutive sentences from the datasets and joined each pair with a space characted. We measured how many times tokenizer correctly splitted a long sentence into two.

tokenizer OpenCorpora SynTagRus
Precision Recall Execution Time (sec) Precision Recall Execution Time (sec)
nltk.sent_tokenize 94.30 86.06 8.67 98.15 94.95 5.07
nltk.sent_tokenize(x, language='russian') 95.53 88.37 8.54 98.44 95.45 5.68
bureaucratic-labs.segmentator.split 97.16 88.62 359 96.79 92.55 210
ru_sent_tokenize 98.73 93.45 4.92 99.81 98.59 2.87

Notebook shows how the table above was calculated

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for rusenttokenize, version 0.0.5
Filename, size File type Python version Upload date Hashes
Filename, size rusenttokenize-0.0.5-py3-none-any.whl (10.5 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size rusenttokenize-0.0.5.tar.gz (5.9 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page