COMBO-SEG

Character-level segmentation model for natural language text.

COMBO-SEG segments raw text into turns, sentences, tokens, and words using a transformer-based character-level classifier. Supports 80+ languages.

Output hierarchy: Document → Turn[] → Sentence[] → Token[] (matches LAMBO). Turns are produced by regexp separators (by default double newline \n\n and <turn>) before the neural model splits each turn into sentences and tokens.

Installation

pip install combo-seg

Usage

from combo_seg import ComboSeg, Language, SplitLevel

segmenter = ComboSeg(language=Language.POLISH)
doc = segmenter("Ala ma kota. Kot ma Alę.\n\nDrugi akapit.")

for turn in doc.turns:
    for sentence in turn.sentences:
        print(sentence.text)
        for token in sentence.tokens:
            print(f"  {token.text}")

# Pass split_level=SplitLevel.TURN to collapse each turn into one Sentence.

License

GPL-3.0