yasbd-lib 0.4.0

A high-accuracy, from-scratch Sentence Boundary Detector (SBD) for production pipelines. Features a drop-in adapter for pysbd to fix edges cases without heavy refactoring.

"Even a pair of scissors deserves to be smart. Welcome to cybernetic boundary shearing."

[!WARNING] This project is currently in alpha.

---

Table of Contents generated with DocToc

• 🎬 Manifesto
  • ✂ Why do I need a pair of "smart scissors" for text?
  • 🔪 Are these shears just a rusty regex loop spray-painted in carbon fiber?
• 🌐 Supported Languages
• 🏁 Benchmarks
• 📦 Installation
  • The Quick & Easy Way
  • The From-Source Way
  • Want to Help Make yasbd Even Better?
• 📟 Usage
  • Initialization
  • Core Methods
    • Boundary detection
    • Segmentation
  • Cleaner
  • CLI
    • About JSONL
  • Adapter
• 🗺 Features & Roadmap
• 🤝 Contributors
• 📜 Last note

---

🎬 Manifesto

Yet Another Sentence Boundary Detector is a pair of smart scissors for text. Pointer-based, from-scratch SBD for production NLP pipelines. Features a drop-in adapter for pysbd to fix edge cases without heavy refactoring.

✂ Why do I need a pair of "smart scissors" for text?

Running re.split(r'\.\s+[A-Z]') and praying. This blunt tool instantly shears titles like Mr. Smith or French corporate markers like Sté. Générale in half, scattering semantic fragments across the pipeline.

Punctuation is the most overloaded glyph set in text. A period alone does six jobs and only one is "sentence end." Generic split-on-punctuation fails on:

• Dr. Inc. U.S.A. (abbreviation markers, not boundaries. ~47% of periods in news text are these)
• 3.5M 3.14 (decimal points, not sentence ends)
• D. H. Lawrence (initials. Two periods, zero boundaries)
• ... (ellipsis. Trailing off or sentence end? ambiguous)
• 1. a. at line start (inline list markers impersonating sentence ends)
• ?! inside quotes (punctuation nesting across boundaries)

And multilingual quirks a naive splitter never saw coming.

🔪 Are these shears just a rusty regex loop spray-painted in carbon fiber?

Regex is how I cut. Not what I am. My brain is a two-pass pipeline:

Pass 1 — Naive boundary finder. Finds every position that could plausibly end a sentence: periods, question marks, exclamation points — anything followed by whitespace, uppercase, or a newline. Deliberately over-inclusive. Better to catch a false positive than miss a real boundary.

Pass 2 — Cross-references 9+ mid-sentence patterns to surgically excise false positives:

• Newline inside sentence
• Title/initialism protection
• Abbreviation lists
• Geopolitical + case markers
• Quote/parenthesis span filtering
• TOC leader suppression
• List marker re-alignmen
• Contiguous terminator collapsing
• Language specifical final fixups

---

🌐 Supported Languages

10 languages supported today. Target is 22+.

	Code	Language
🇸🇦	ar	Arabic
🇩🇪	de	German
🇬🇧	en	English
🇪🇸	es	Spanish
🇫🇷	fr	French
🇭🇹	ht	Haitian Creole
🇯🇵	ja	Japanese
🇵🇹	pt	Portuguese
🇷🇺	ru	Russian
🇨🇳	zh	Chinese

You can also get a list from yasbd.get_supported_langs.

---

🏁 Benchmarks

Tested against 6 competitors (pysbd, sentencex, sentsplit, nupunkt, blingfire, sentence-splitter) across 5 languages and 7 edge cases: compound abbreviations, CJK quotes, newline wrapping, chat logs, URLs, decimals, and nested punctuation.

TL;DR: yasbd ranked #1 in accuracy across almost every test, while staying competitive on speed as pure Python. blingfire is faster but brittle. pysbd and sentencex shred French abbreviations.

On our golden benchmark (85 English edge cases adapted from pysbd's official golden rule set with fixes and additions): yasbd scores 100.0%, pysbd 84.7%.

Full results, terminal output, and a performance graph can be found in benchmarks/

---

📦 Installation

Ready to do some cybernetic boundary shearing? Let's get you set up quickly and painlessly.

The Quick & Easy Way

The simplest way to get started is with pip:

pip install yasbd-lib -U

[!TIP] Termux (Android)

No Rust toolchain? Install pydantic-core pre-built wheels first, then retry:

pip install typing-extensions
pip install pydantic-core --index-url https://termux-user-repository.github.io/pypi/
pip install "pydantic>=2.12.4,<2.13"

That's it! Blade is armed.

The From-Source Way

Prefer building from source? Clone and install manually for full control:

git clone https://github.com/speedyk-005/yasbd-lib.git
cd yasbd
pip install .

(But honestly, the pip way is way easier.)

Want to Help Make yasbd Even Better?

That's awesome. See Contributing Guide.

---

📟 Usage

[!TIP] Not a Pythonista? Jump straight to the CLI section.

Looking for the pysbd drop-in replacement? Jump straight to the Adapter section.

Initialization

from yasbd.boundary_detector import BoundaryDetector
# Or from yasbd import BoundaryDetector

# Basic setup
detector = BoundaryDetector(lang="en")

# With all options (so far.)
detector = BoundaryDetector(
	# ISO 639 code (e.g., en, fr, es, ...). Defaults to `en`.
	# https://en.wikipedia.org/wiki/List_of_ISO_639_language_codes
    lang="fr",

    # Don't split inside them. (It won't protect block quotes) Defaults to `True`.
    # https://en.wikipedia.org/wiki/Block_quotation
    preserve_quote_and_paren=True,

    # Enable verbose logging. Defaults to `False`.
    verbose=True,
)

Switching languages at runtime is a property set:

detector.lang = "es"

The rule module loads lazily on first access. Switching mid-stream reimports the module and rebinds the pattern cache. Zero config, no restarts needed.

Core Methods

The two primary APIs are detect() and segment().

Both methods accept plain strings, open text streams (TextIOBase), or a StreamCleaner instance. Inputs are processed lazily as a stream of paragraphs, allowing large documents to be handled without loading everything into memory at once.

• detect() yields sentence boundary offsets.
• segment() yields sentence strings.

Boundary detection

detect() tells you where each sentence stops. Integer offsets into the original input stream.

Two detection modes:

• absolute: (default) offsets count from the start of the entire input stream.
• relative: offsets reset at each paragraph boundary. A ParagraphEOF sentinel signals the gap between paragraphs.

# absolute mode (default)
res= list(detector.detect('She turned to him, "This is great." She held the book out to show him.'))
print(res)
# [35, 70]

# relative mode with paragraph break
detector.lang = "es"
res = list(detector.detect(
	"El Sr. García llegó ayer. La Sra. López también.\n\nVéase la pág. 55 del libro.",
	relative=True,
))
print(res)
# [25, 48, ParagraphEOF, 27]

Segmentation

If you do not want to manage boundary offsets yourself (and who would?), segment() slices text for you.

detector.lang = "en"

# Basic sentence splitting
res = list(detector.segment("Hello world. How are you? I am fine."))
print(res)
# ['Hello world.', 'How are you?', 'I am fine.']

# Multi-paragraph with whitespace preserved
res = list(detector.segment(
    "First para.\nStill first.\n\nSecond para.\nFinished.",
    preserve_whitespace=True,
))
print(res)
# ['First para.', '\nStill first.', '\n\n', 'Second para.', '\nFinished.']

[!TIP] ParagraphStream — yasbd uses ParagraphStream internally to split text into paragraph blocks. You can import it directly if you need paragraph-level processing in your own code:

from yasbd.utils.paragraph_stream import ParagraphStream  # or yasbd.paragraph_stream

for para in ParagraphStream(text):  # or an opened file
    print(para)  # each paragraph block

You can also skip empty lines with skip_empty_lines=True

Cleaner

OCR'd a PDF, parsed a DOCX, or scraped noisy HTML? "StreamCleaner" normalizes text before it reaches the language detector or sentence segmenter. StreamCleaner accepts either a string or an open text stream and yields cleaned paragraphs lazily. You can pass a "StreamCleaner" instance directly to "detect()" or "segment()" to clean text as it is processed.

from yasbd.utils.cleaner import StreamCleaner  # or yasbd.cleaner

cleaner = StreamCleaner(
    "Hello  world.   This is  messy.",
    verbose=True,  # Default to False
)
list(cleaner)
# ['Hello world. This is messy.']

"StreamCleaner" implements the iterator protocol and yields cleaned paragraphs one at a time. It can consume plain strings, open text files, and other text streams.

from yasbd.utils.cleaner import StreamCleaner  # or yasbd.cleaner

with open("document.txt", encoding="utf-8") as f:
    for paragraph in StreamCleaner(f):
        print(paragraph)

Common cleanup operations include:

• Fixing mojibake and OCR artifacts
• Removing HTML tags
• Normalizing whitespace and repeated slashes
• Rejoining hyphenated words split across lines
• Merging vertically stacked characters

Skip built-in steps you don't want:

cleaner = StreamCleaner(
    text,
    steps_to_skip=[
        "fix_ocr_text",
        "normalize_spaces",
    ],
)

Add custom cleaning steps:

cleaner = StreamCleaner(
    text,
    extra_steps=[
        lambda t: t.replace("™", ""),
        lambda t: t.upper(),
    ],
)

Each extra step must accept and return a str. If a step raises or returns a non-string, a CleanStepError is raised with the original exception chained.

Available built-in steps:

Step	What it does
fix_mojibake	Fixes Unicode mojibake via ftfy
fix_ocr_text	Repairs OCR artifacts, rejoins hyphenated words, removes page markers
unwrap_htmls	Strips HTML tags (including <script>, <style> and their content)
normalize_slashes	Collapses /// triple slashes
normalize_spaces	Collapses multiple spaces into one

CLI

Do you just want to split text into sentences without writing Python? The yasbd command works right from your terminal. Install once, pipe anything into it, get sentences back.

# List supported language codes
yasbd langs
# ar, de, en, es, fr, ht, ja, pt, ru, zh

# Split text into sentences
yasbd segment "Dr. Smith works here. Is he there?"
# [1] 'Dr. Smith works here.'
# [2] 'Is he there?'

# Detect boundary offsets
yasbd detect "Hello world. How are you?"
# [1] 12
# [2] 24

# Read from file
yasbd segment --file document.txt
yasbd segment --file input.txt --destination output.txt  # JSONL output

# Pipe support — auto-detects, skips [N] enumeration
echo "Hello. World." | yasbd segment | cat
# Hello.
# World.

# Clean noisy text (HTML, mojibake, OCR artifacts)
yasbd clean "<script>x</script>Hello <b>world</b>."
# [1] 'Hello <b>world</b>.'

# Clean with extra shell command step (e.g., transliterate via external tool)
yasbd clean "naïve café" --extra-step "sed 's/é/e/g; s/ï/i/g'"
# [1] 'naive cafe'

# Repeatable --extra-step for multiple shell commands
yasbd clean "HELLO." -e "tr 'A-Z' 'a-z'" -e "sed 's/\./!/g'"
# [1] 'hello!'

# Chaining: Skip HTML unwrap then segment in Spanish with verbosity
yasbd clean --file dirty.html --skip unwrap_htmls | yasbd segment --lang es -v

# Version
yasbd --version

# Full help
yasbd --help         # top-level commands
yasbd segment --help # per-command options
yasbd detect --help
yasbd clean --help

CLI API reference at API_REFERENCES.md#yasbd.cli.

About JSONL

When writing to a file with --destination, output is JSONL (one JSON object per line):

• segment / clean: {"no": 1, "text": "Hello."}
• detect: {"no": 1, "offset": 6} or {"no": 2, "offset": 13}
• detect --relative: {"no": 3, "eof": true} on paragraph boundaries

Adapter

Migrating from pysbd? Swap the import and keep your pipeline:

# Before: from pysbd import Segmenter
from yasbd.utils.pysbd_adapter import Segmenter  # or yasbd.pysbd_adapter

seg = Segmenter(language="ja")
res = seg.segment('田中さんは「準備は完了しました」そう言って部屋を出た。Ｕ．Ｓ．Ａ．の経済政策は非常に複雑です。')
print(res)
# ['田中さんは「準備は完了しました」そう言って部屋を出た。', 'Ｕ．Ｓ．Ａ．の経済政策は非常に複雑です。']

Same API surface. Same Segmenter class. Same segment() method signature.

---

🗺 Features & Roadmap

• Regex caching (compile once per language class)
• Drop-in pysbd adapter (same API, no pipeline changes)
• StreamCleaner for OCR'd and noisy text
• CLI tool
• spaCy integration
• 22+ language targets
• REST API for remote boundary detection

---

🤝 Contributors

A massive thank you to the open source community helping make yasbd more accurate and scalable:

Name	Role
@speedyk-005	Maintainer & Creator
@JheanLL	Trie prototype design & Spanish rule contributions
@Jah-yee	Community contributor
@Rajesh270712	Base + English rule contributions

Interested in contributing? See the Contributing Guide to get started!

---

📜 Last note

yasbd is maintained by speedyk-005. Licensed under Mozilla Public License 2.0 — you can use it freely in commercial and private work.

Star us on GitHub if you dig it. Tell your NLP pipeline we said hi. 🚀