segments

The segments package provides Unicode Standard tokenization routines and orthography segmentation, implementing the linear algorithm described in the orthography profile specification from The Unicode Cookbook (Moran and Cysouw 2018 ).

Command line usage

Create a text file:

$ echo "aäaaöaaüaa" > text.txt

Now look at the profile:

$ cat text.txt | segments profile
Grapheme        frequency       mapping
a       7       a
ä       1       ä
ü       1       ü
ö       1       ö

Write the profile to a file:

$ cat text.txt | segments profile > profile.prf

Edit the profile:

$ more profile.prf
Grapheme        frequency       mapping
aa      0       x
a       7       a
ä       1       ä
ü       1       ü
ö       1       ö

Now tokenize the text without profile:

$ cat text.txt | segments tokenize
a ä a a ö a a ü a a

And with profile:

$ cat text.txt | segments --profile=profile.prf tokenize
a ä aa ö aa ü aa

$ cat text.txt | segments --mapping=mapping --profile=profile.prf tokenize
a ä x ö x ü x

API

>>> from segments import Profile, Tokenizer
>>> t = Tokenizer()
>>> t('abcd')
'a b c d'
>>> prf = Profile({'Grapheme': 'ab', 'mapping': 'x'}, {'Grapheme': 'cd', 'mapping': 'y'})
>>> print(prf)
Grapheme	mapping
ab	x
cd	y
>>> t = Tokenizer(profile=prf)
>>> t('abcd')
'ab cd'
>>> t('abcd', column='mapping')
'x y'

Metadata

Release files for segments 2.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for segments 2.4.0
File	Size	Uploaded
segments-2.4.0.tar.gz	18.7 kB	Mar 7, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for segments 2.4.0
File	Interpreter	ABI	Platform	Reset
segments-2.4.0-py2.py3-none-any.whl 16.3 kB Mar 7, 2026	Python 2, Python 3	none	any	Details

Total release size: 35.0 kB

Release files / segments-2.4.0.tar.gz

Download URL	segments-2.4.0.tar.gz
Size	18.7 kB
Tags	Source
SHA-256 checksum How to use checksums	`bba71f5520ddd54c8aa2f4d765a60618c6862162d6e7356a4a097f2223166f5b`
BLAKE2b-256 checksum How to use checksums	`8b5785cac3a8e32370e88fa5fa92812edb6025db7fcbed51452bd56ee1524957`
Upload date	Mar 7, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.3`

Release files / segments-2.4.0-py2.py3-none-any.whl

Download URL	segments-2.4.0-py2.py3-none-any.whl
Size	16.3 kB
Tags	Python 2 Python 3
SHA-256 checksum How to use checksums	`4021dc67f201cc03c864c74c618bdb163b1af629da3040babbaa37d8813f3db0`
BLAKE2b-256 checksum How to use checksums	`be60eef9acce946177f92c9aabf432224d87ab908bafafac516a36ab924199f3`
Upload date	Mar 7, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.12.3`