NLP Text Preprocessing Utility Package

Table of Contents

• Package Overview
• Key Features
• Architecture Diagram
• Quickstart and Setup
• Pipeline Automation Engine
• Complete API Reference
  • 1. General Feature Extraction
  • 2. Text Cleaning and Normalization
  • 3. Linguistic and Morphological Processing
  • 4. Safety, Moderation and Sentiment
  • 5. Visualization Suite
• Code Examples
  • Basic Text Cleaning
  • Pipeline Engine Usage
  • Pandas DataFrame Integration
  • Batch Feature Extraction
  • WordCloud Generation
• Troubleshooting and Resource Setup
• License and Author

Package Overview

nlp_text_preprocessing is a production-grade, modular Python library designed to streamline text cleaning, feature extraction, linguistic analysis, content moderation, and visualization for Natural Language Processing (NLP) pipelines and Machine Learning workflows.

Maintained by Uditya Narayan Tiwari.

It provides over 60 single-purpose functions and an extensible Pipeline Engine with built-in defensive guards for None, NaN, and missing values, making it 100% crash-proof when running over Pandas DataFrames or large raw text datasets.

Key Features

• Crash-Proof Pipeline: Automatic None/NaN input sanitation across all functions.
• Pipeline Automation Engine: Chain cleaning steps by name or custom functions via Pipeline(['to_lower_case', 'remove_urls', 'lemmatize']).
• Lazy and Fast Model Caching: Efficient lazy-loading for spaCy (en_core_web_sm) and cached NaiveBayes sentiment analyzers.
• Advanced Cleaning: Remove emojis, phone numbers, PII mask, URLs, emails, HTML, retweets, special symbols, and expand contractions/hashtags.
• Feature & Readability Extraction: Extract word counts, sentence counts, readability scores (Flesch Ease), punctuation density, capital ratios, and full batch statistics.
• Linguistic Utilities: Named Entity Recognition (NER), POS tagging, dependency parsing, language detection, keyword extraction, synonyms, antonyms, and n-grams.
• Safety & Moderation: Profanity detection/censoring, toxicity scores, and spam signal detection.
• Visualization: Generate and save custom WordClouds to disk or display interactively.

Architecture Diagram

The package integrates spaCy, NLTK, TextBlob, BeautifulSoup4, and WordCloud through a unified API surface:

Quickstart and Setup

1. Installation

pip install nlp-text-preprocessing

2. Downloading Required Corpora and Models

To automatically download all required NLTK corpora (stopwords, movie_reviews, brown) and spaCy models (en_core_web_sm), run:

import nlp_text_preprocessing as tp

# One-time automated setup for NLTK and spaCy resources
tp.download_nltk_packages()

Pipeline Automation Engine

The Pipeline class allows chaining cleaning and transformation steps cleanly:

import nlp_text_preprocessing as tp

# 1. Create default zero-config pipeline
pipe = tp.Pipeline.default()

# 2. Or build a custom step pipeline
pipe = tp.Pipeline([
    'to_lower_case',
    'expand_hashtags',
    'remove_urls',
    'remove_emails',
    'remove_extra_whitespace',
    'lemmatize'
])

# Process single text
clean_result = pipe.run("Check out #MachineLearning at https://example.com!")

# Process batch list or Pandas Series
clean_batch = pipe.run_batch(["Text 1", "Text 2"])
clean_series = pipe.run_series(df['text_column'])

# Generate diff report of changes
report = tp.diff_report(raw_texts, clean_batch)

Complete API Reference

Below is the complete reference guide for all functions available in the library:

1. General Feature Extraction

2. Text Cleaning and Normalization

3. Linguistic and Morphological Processing

4. Safety, Moderation and Sentiment

5. Visualization Suite

Code Examples

1. Basic Text Cleaning

import nlp_text_preprocessing as tp

raw_text = "Check out https://example.com! Contact support@company.org or RT @user 'I'm loving #NLP'."
cleaned = tp.clean_text(raw_text)

print(cleaned)
# Output: check out contact support company org or i am love nlp

2. Pipeline Engine Usage

import nlp_text_preprocessing as tp

# Build custom pipeline
pipe = tp.Pipeline(['to_lower_case', 'expand_hashtags', 'remove_urls', 'remove_extra_whitespace'])

text = "Check out #MachineLearning at https://example.com!"
print(pipe.run(text))
# Output: check out machine learning at!

3. Pandas DataFrame Integration

All functions seamlessly support Pandas .apply() and handle missing values (NaN/None):

import pandas as pd
import nlp_text_preprocessing as tp

df = pd.DataFrame({
    'text': [
        "I'm loving this NLP package! #awesome",
        "Contact me at info@test.com or visit https://test.com",
        None
    ]
})

# Apply end-to-end text cleaning
df['clean_text'] = df['text'].apply(tp.clean_text)

# Extract word counts safely
df['word_count'] = df['text'].apply(tp.word_count)

4. Batch Feature Extraction

Extract full feature matrices directly into Pandas columns:

import pandas as pd
import nlp_text_preprocessing as tp

df = pd.DataFrame({'text': ["Hello #world! Visit https://example.com", "Contact user@test.com"]})

# Extract all summary stats into a new DataFrame
stats_df = tp.text_stats_batch(df, 'text')
print(stats_df)

5. WordCloud Generation

import nlp_text_preprocessing as tp

text = "python natural language processing machine learning text mining data science spaCy nltk textblob"

# Save directly to disk without opening interactive GUI window
tp.get_wordcloud(text, save_path="wordcloud.png", show=False)

Troubleshooting and Resource Setup

If any NLTK corpus or spaCy model throws a missing resource error during runtime, simply run:

import nlp_text_preprocessing as tp
tp.download_nltk_packages()

Or manually download spaCy model via terminal:

python -m spacy download en_core_web_sm

License and Author

• Author: Uditya Narayan Tiwari
• License: MIT License (LICENSE)