anySeg

Any Segmentation - fast segmentation of text into tokens and sentences, written in Rust with a Python API

Documentation · PyPI · crates.io

anySeg segments text into tokens, words and sentences with their positions in the source text: a Rust core and a Python package without dependencies. The text is never changed - every token and sentence is a span of the original string - and the tokens follow the conventions of Universal Dependencies, so the results can be compared with the treebanks and with the tools trained on them. One package serves all languages: the rules are shared, and a language brings its lists. Russian comes first, Spanish and English follow.

From Python, tokens and sentences come out at 3.1 M tokens per second, six times faster than razdel; the Rust crate gives 13.4 M. On the test files of the Russian treebanks of UD, token F1 is 99.1-99.8, above every other tool on Taiga, GSD and PUD, and the sentence boundaries are above razdel on all four sets.

• Tokens - words with stress marks and soft hyphens, hyphenated words by the UD guidelines, numbers, dates and times, URLs, e-mail addresses, mentions, hashtags, emoticons and emoji, abbreviations and initials with their period
• Sentences - final marks with the quotes and brackets after them, abbreviations that end a sentence, lines of dialogue, list items, headings and verse
• Paragraphs and lines - paragraphs between blank lines, and every line marked as a break of the author or a wrap at the width of the page, as in text from PDF
• Document model - tokens, words with the columns of CoNLL-U, sentences, paragraphs and lines; reading and writing CoNLL-U, a document read from it written back as it was
• Lists of your own - more abbreviations and strings that are one token each
• Command - anyseg for the terminal and pipelines: tokens, sentences or CoNLL-U of files and the standard input
• spaCy - the tokenizer and the sentence starts of a spaCy pipeline, the conversion of documents
• Comparison - quality and speed against razdel, NLTK, spaCy, Stanza, trankit, SaT and other tools

Installation

Python 3.11 or newer is required.

pip install anyseg

Or with uv:

uv add anyseg

The tokenizer and the sentence starts for spaCy come with the extra spacy: pip install "anyseg[spacy]".

Wheels are built for Linux (Alpine included), macOS and Windows, the free-threaded Python 3.14t and 3.15t included; installing them does not require Rust. On other platforms pip builds the package from source, which requires Rust 1.88 or newer. The segmenter is also the Rust crate anyseg. See the Installation page.

Quick start

>>> from anyseg import Segmenter, tokenize

>>> segmenter = Segmenter("ru")
>>> [span.text for span in segmenter.sentenize("Кот спал в 1914 г. Затем ушёл, т. е. исчез.")]
['Кот спал в 1914 г.', 'Затем ушёл, т. е. исчез.']
>>> segmenter.tokenize("Ёж идёт домой")
[Span(start=0, end=2, text='Ёж'), Span(start=3, end=7, text='идёт'), Span(start=8, end=13, text='домой')]
>>> [span.text for span in tokenize("Ростова-на-Дону, кто-то, 2,5-3,5 млн", lang="ru")]
['Ростова', '-', 'на', '-', 'Дону', ',', 'кто-то', ',', '2,5', '-', '3,5', 'млн']

Create the segmenter once and use it throughout the program: it is immutable, can be shared between threads and pickled. Positions are offsets in the characters of a Python string, text[span.start:span.end] == span.text; the Rust crate gives them in bytes.

A document holds the tokens, the words with the columns of CoNLL-U, the sentences, the paragraphs and their lines:

>>> doc = segmenter.segment("Ёж идёт домой\n\nКот спит")
>>> [sentence.text for sentence in doc.sentences]
['Ёж идёт домой', 'Кот спит']
>>> doc.set_columns(upos=["NOUN", "VERB", "ADV", "NOUN", "VERB"])
>>> doc.words[1]
Word(id=2, form='идёт', upos='VERB')

From the terminal, anyseg -l ru -f conllu text.txt writes the text in CoNLL-U. segmenter.pipe(texts, threads=4) gives the documents of many texts, segmented in batches without the GIL on several threads. doc.to_conllu() writes a document in CoNLL-U, and read_conllu(text) reads documents from it. A segmenter takes more abbreviations and strings that are one token each: Segmenter("ru", abbreviations=["сущ"], tokens=["C++"]).

Comparison

On the test files of the Russian treebanks of Universal Dependencies 2.18, F1 (* marks neural tools trained on SynTagRus):

From Rust, the crate gives 13.4 M tokens per second with sentences. All tools and sets, and how it is measured: Comparison.

Development

The project uses uv to manage dependencies, maturin to build the Rust module and ruff for linting and formatting. Rust 1.88 or newer is required.

git clone https://github.com/SergeyShk/anySeg.git
cd anySeg

make deps        # create the environment, install the dependencies and build the Rust module
make test        # run the Python tests, the examples of the docstrings, the README and the documentation, and the Rust tests
make lint        # ruff, mypy, stubtest, rustfmt and clippy

The full list of commands is make help. Before submitting changes, install the hooks that run the linters on commit and the tests on push:

uv run pre-commit install

Contributing

Bug reports, ideas and pull requests are welcome - the issues are open. The workflow and the checks to run before submitting a pull request are in CONTRIBUTING.md; the rules of conduct are in the code of conduct.

Authors

• Sergey Shkarin (kouki.sergey@gmail.com)

License

The code and the lists of the languages are distributed under the MIT license.

Citation

If you use anySeg in research or software, please cite it with the following BibTeX entry - citations help the development and support of the project. The same metadata is in CITATION.cff, which GitHub shows under the "Cite this repository" button. The concept DOI 10.5281/zenodo.23788634 on Zenodo points to all versions of the package; the DOI of a version is on the page of its release.

@software{anySeg,
  author = {Sergey Shkarin},
  title = {{anySeg, fast segmentation of text into tokens and sentences}},
  year = 2026,
  doi = {10.5281/zenodo.23788634},
  url = {https://github.com/SergeyShk/anySeg}
}