cleave-sbd: Sentence Boundary Disambiguation

cleave-sbd is a high-performance, strictly-typed sentence boundary disambiguation (SBD) engine. It isolates sentence boundaries across abbreviations, honorifics, numbers, lists, ellipses, and quotations without machine learning dependencies.

Features

• Zero Heavy Dependencies: Pure Python logic without neural models, PyTorch, or GPU requirements.
• Length-Preserving Coordinate Mapping: PUA sentinel substitutions and offset delta tracking ensure 1:1 character offset invariance for precise span extraction.
• Strict Typing: Fully typed and verified in strict mode with Basedpyright/Pyright (PEP 561 compliant with py.typed).
• Multilingual Support: Out-of-the-box rule sets for 22 languages.
• High Performance: Pre-compiled regular expressions and immutable lookup tables.

Installation

pip install cleave-sbd

Or with uv:

uv add cleave-sbd

Quickstart

import csbd

text = "My name is Jonas E. Smith. Please turn to p. 55."
seg = csbd.Segmenter(language="en", clean=False)

sentences = seg.segment(text)
print(sentences)
# Output:
# ('My name is Jonas E. Smith.', 'Please turn to p. 55.')

Character Span Mode

Extract start and end character offsets alongside segmented sentences:

import csbd

text = "Hello world! This is a test."
seg = csbd.Segmenter(language="en", char_span=True)

spans = seg.segment(text)
for span in spans:
    print(f"{span.sent!r} -> [{span.start}:{span.end}]")
# Output:
# 'Hello world!' -> [0:12]
# 'This is a test.' -> [13:28]

Streaming Mode

Lazily process large texts or documents paragraph-by-paragraph with bounded memory:

import csbd

seg = csbd.Segmenter(language="en")
for sentence in seg.stream(large_text, chunk_paragraphs=1000):
    print(sentence)

Parallel Processing (Multi-Worker Safe) (Coming Soon)

Native multi-core batch processing leveraging pure immutable state machines across worker processes.

Parameters

Supported Languages

Why cleave-sbd?

Documentation & Deep Dives

For complete documentation, architectural specifications, and recipes, see the cleave-sbd Wiki:

• 1. Design Philosophy: The 10 design rules behind our zero-dependency build, strict typing, and memory model.
• 2. Span Tracking: The mechanics behind our PUA masks and $O(\log K)$ delta tracking for exact character offsets.
• 3. Use Cases: Code recipes for sentence segmentation, span extraction, bounded streaming, and data cleaning.
• 4. Languages: How we handle abbreviations, legal numbering, non-Latin scripts, and unspaced text across 22 languages.
• 5. Benchmarks: Benchmark comparisons, latency tables, and failure analysis against Stanza, spaCy, NLTK, BlingFire, Syntok, and pySBD.
• 6. Contributing: Developer guide for running automated quality gates (validate.sh) and adding language configurations.

Acknowledgments & Attribution

cleave-sbd is an independent, complete rewrite designed from the ground up as a modern, declarative, strictly-typed sentence boundary disambiguation engine.

Sincere attribution and gratitude are given to the projects whose compiled linguistic heuristics and rule sets inspired this library:

• Pragmatic Segmenter by Kevin S. Dias (Ruby)
• pySBD by Nipun Sadvilkar (Python)

License

MIT License. See LICENSE for details.