tokenizesentences

Tiny, dependency-free English sentence tokenizer.

installation

uv add tokenizesentences

or

pip install tokenizesentences

usage

In [1]: from tokenizesentences import tokenize

In [2]: tokenize(
    "Mr. John Johnson Jr. was born in the U.S.A but earned his Ph.D. in Israel before joining Nike Inc. as an engineer. He also worked at craigslist.org as a business analyst."
    )

Out[2]:
[
    'Mr. John Johnson Jr. was born in the U.S.A but earned his Ph.D. in Israel before joining Nike Inc. as an engineer.',
    'He also worked at craigslist.org as a business analyst.'
]

Sentence offsets are available too. Every span is a half-open [start, end) slice of the input, so text[start:end] is the sentence, verbatim:

In [3]: from tokenizesentences import tokenize_spans

In [4]: tokenize_spans("It works. Try it!")
Out[4]: [(0, 9), (10, 17)]

notes

Deterministic, rule-based and English-only: an index-based scanner inspired by the answer of D Greenberg in StackOverflow, with no models and no dependencies.

Sentences are literal substrings of the input, by construction. The text is never rewritten: punctuation stays where the author put it (also inside quotes) and single newlines inside a sentence are preserved. A blank line always ends a sentence.

The heuristics are conservative: when in doubt they join, so a missed boundary is preferred over a spurious split. Known limits, pinned by the test suite:

• A lowercase word after the mark never splits: she yelled "Stop!" and ran away. is one sentence, and so is informal text like i came. i saw.
• A dot glued to a word never splits (that is how any domain, filename or version number survives without a TLD list), so the typo He left.Then she cried. stays joined.
• ... (glued or spaced . . .) ends a sentence only when a capitalized word follows. I does not count: it is always capitalized, so it carries no signal.
• After ambiguous abbreviations (p.m., Inc., U.S.A.) a sentence break is only detected before common starter words: at 5 p.m. He left splits, at 5 p.m. Monday does not.
• Titles (Mr., Sen., St.) never end a sentence, so Main St. He walked. stays joined.

why tokenizesentences?

tokenizesentences is designed for pipelines where sentence splitting should be a small, predictable primitive rather than a heavyweight NLP component. It is conservative, fast, dependency-free and lossless:

• Conservative: ambiguous boundaries are joined rather than split, favoring fewer false boundaries over higher recall.
• Fast: a small index-based scanner keeps segmentation cheap in high-volume pipelines.
• Dependency-free: no models, no data downloads, no runtime dependencies.
• Lossless: sentences are literal slices of the original text, with exact offsets available through tokenize_spans().

That combination is built for the case where a false split costs more than a missed boundary: preprocessing, scraping, indexing, annotation, highlighting, and NLP or LLM pipelines where preserving the original text and its offsets matters more than squeezing out the last point of F1.

benchmarks

• 40 of the 52 English Golden Rules from pragmatic_segmenter pass verbatim; the other 12 are pinned in tests/test_golden_rules.py as strict xfails with documented reasons (list detection, text rewriting, ellipsis attachment, glued sentences).

• Boundary detection on the UD English-EWT test split: 2,077 gold sentences of raw web text in 316 reconstructed documents, scored on the 1,761 interior boundaries (the trivial final boundary of each document is excluded). All systems ran the same day (August 2026) on the same inputs with the same scoring; reproduce with uv run python tools/eval_ewt.py --competitors --speed:

  system	precision	recall	F1	throughput
  pysbd	95.67%	84.10%	89.51%	0.05 Mchar/s
  tokenizesentences	99.50%	79.22%	88.21%	4.6 Mchar/s
  syntok	98.38%	79.16%	87.73%	0.30 Mchar/s
  nltk-punkt	97.38%	69.56%	81.15%	1.91 Mchar/s
  spacy-sentencizer	96.62%	68.14%	79.92%	0.60 Mchar/s
  blingfire	98.79%	60.02%	74.67%	1.50 Mchar/s

  Highest precision and throughput of the six on this evaluation; only pysbd scores a higher F1, trading 3.8 points of precision and two orders of magnitude of throughput for it. Recall is spent on informal lowercase web text on purpose. Timings are medians of 7 passes after a warm-up; the blingfire and syntok figures include offset reconstruction overhead. The corpus commit, its SHA256 and the competitor versions are pinned, so the run is reconstructible: see tools/eval_ewt.py and the benchmark dependency group.

• The English suite of Wikimedia's sentencex (MIT), pinned at an exact commit and SHA256, with its parsing and per-case multiset-F1 metric replicated verbatim (234 scoreable cases; the 60-case table in their README predates their current suite). Same-day run, August 2026; reproduce with uv run python tools/eval_sentencex.py --competitors:

  system	mean F1	perfect cases
  sentencex	99.94	233/234
  tokenizesentences	86.17	195/234
  blingfire	83.08	188/234
  syntok	80.79	180/234
  pysbd	64.33	136/234
  nltk-punkt	62.92	132/234
  spacy-sentencizer	62.52	134/234

  Second only to sentencex itself on sentencex's home suite, first among everything else. The ranking shuffle against the EWT table above (pysbd drops from first to fifth) is exactly why this README carries more than one benchmark.

development

uv sync --group dev
uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run ty check

license

This project is licensed under the MIT License.