SubSegmenter

A lightweight Python library built to segment dense text down to atomic semantic units. Unlike traditional tokenizers (like nltk) that stop strictly at the sentence level, SubSegmenter resolves shared dependencies and breaks sentences apart at clausal and gerund breakpoints without losing context words or language.

Core Features

• Shared Dependency Resolution: Expands coordinated structures smoothly (e.g., "design and lead projects" becomes "design projects", "lead projects").
• High-Performance Execution: Streamlined to run entirely on the CPU.
• Fast Text Pre-Cleaning: Built-in optimized regex filters to strip HTML markup, metadata, and other junk before parsing.
• Domain-Agnostic: Runs completely based on linguistic parsing.

Installation

pip install subsegmenter

Also make sure to have the appropriate SpaCy model downloaded:

python3 -m spacy download en_core_web_sm

Quick Start

Processing any document is easy:

from subsegmenter import SubSegmenter

text = "Organizes, coordinates and mentors the efforts of project engineers supporting the program. Maintains consistency with applicable standards, procedures, and implementation methodologies."

segmenter = SubSegmenter()
results = segmenter.segment(text)

If you would like to process a large batch of (large) documents, use the following:

from subsegmenter import process_pipeline

documents = [text] * 1000

batch_outputs = process_pipeline(documents)