DSTPR: Domain-specific Text Pre-processing and Ranking

A high-throughput text preprocessing pipeline designed to filter, segment, and rank core sentences from noisy, plaintext documents.

This package strips away boilerplate, disclaimers, and application instructions before moving this data onto to heavier processing pipelines.

Pipeline Architecture

Instead of running heavy transformer models over every sentence in a document, DSTPR pipes documents through progressively stricter layers:

1. Heuristic Cleansing & Tokenization: Uses PySBD (Python Sentence Boundary Disambiguation) paired with regular expressions to fix punctuation caused by flattening, then filters out sentences containing stop phrases.
2. Semantic Section Routing: Utilizes a lightweight encoder (Default: all-MiniLM-L6-v2) to dynamically find structural transitions (i.e., headers) in input texts.
3. Hybrid Ranking: Scores remaining sentences using a combination of Semantics (cosine similarity to anchors) and Lexical Syntax (regex- and POS-supported detection).
4. Parallel Execution Engine: Wraps the entire pipeline inside efficient datasets to allow for batch processing at scale, with CPU or GPU.

Installation

To install the package, simply run

pip install dstpr

Usage

High-Throughput Batch Processing

This is the recommended approach for large-scale data pipelines.

from dstpr import ParallelPreprocessingPipeline, JOB_POSTING_PROFILE

pipeline = ParallelPreprocessingPipeline(profile=JOB_POSTING_PROFILE, batch_size=64)

raw_documents = [
    "DOC 1",
    "DOC 2",
    "..."
]

cleaned_documents = pipeline.process(raw_documents, num_workers=8, threshold=0.25)

Advanced Usage

If you want to integrate specific pipeline layers directly into an existing workflow, or tweak the internal parameters, you can import individual modules manually:

from dstpr.cleaners import clean_and_split_chunks
from dstpr.segmenters import SemanticSectionRouter
from dstpr.rankers import HybridRanker
from sentence_transformers import SentenceTransformer

embedding_model = SentenceTransformer("all-MiniLM-L6-v2")

text = "Job posting text goes here!"

# Clean and segment sentences
sentences = clean_and_split_chunks(text)

# Route by context
router = SemanticSectionRouter(model=embedding_model)
buckets = router.route_sentences(sentences)
target_sentences = buckets['CORE'] + buckets['REQUIREMENTS']

# Grade and sort text features
ranker = HybridRanker()
final = ranker.rank_and_filter(target_sentences, bi_cutoff_pct=0.25, final_threshold=0.40)

Parameter Tuning

To adjust the trade-off between strict filtering and execution speed, consider tweaking these variables in ParallelPreprocessingPipeline:

• threshold (Default: 0.4): Controls how aggressively sentences are discarded. Raising this value toward 0.6, for example, ensures only stronger matching sentences are passed through. Lowering it toward 0.3, on the other hand, acts as a wider net.
• batch_size (Default: 256): This depends on your hardware! Adjust for best performance.

Creating Custom Domain Profiles (via the provided wizard)

A DomainProfile is required to use DSTPR (see the usage example). Out of the box, dstpr ships with pre-configured configurations for job postings (JOB_POSTINGS_PROFILE). However, you can easily generate an domain-specific pipeline for any specific domain using our built-in interactive configuration wizard.

To spin up the profile creation walkthrough, simply open your terminal and run:

task-profile-wizard

All profiles generated via the terminal wizard are automatically validated and written out as JSON to a local cache: ~/.config/dstpr/profiles/