Skip to main content

A python package for text preprocessing task in natural language processing.

Usage

To use this text preprocessing package, first install it using pip:

pip install text-preprocessing

Then, import the package in your python script and call appropriate functions:

from text_preprocessing import preprocess_text
from text_preprocessing import to_lower, remove_email, remove_url, remove_punctuation, lemmatize_word

# Preprocess text using default preprocess functions in the pipeline
text_to_process = 'Helllo, I am John Doe!!! My email is john.doe@email.com. Visit our website www.johndoe.com'
preprocessed_text = preprocess_text(text_to_process)
print(preprocessed_text)
# output: hello email visit website

# Preprocess text using custom preprocess functions in the pipeline
preprocess_functions = [to_lower, remove_email, remove_url, remove_punctuation, lemmatize_word]
preprocessed_text = preprocess_text(text_to_process, preprocess_functions)
print(preprocessed_text)
# output: helllo i am john doe my email is visit our website

Features

Feature

Function

convert to lower case

to_lower

convert to upper case

to_upper

keep only alphabetic and numerical characters

keep_alpha_numeric

check and correct spellings

check_spelling

expand contractions

expand_contraction

remove URLs

remove_url

remove names

remove_name

remove emails

remove_email

remove phone numbers

remove_phone_number

remove SSNs

remove_ssn

remove credit card numbers

remove_credit_card_number

remove numbers

remove_number

remove bullets and numbering

remove_itemized_bullet_and_numbering

remove special characters

remove_special_character

remove punctuations

remove_punctuation

remove extra whitespace

remove_whitespace

normalize unicode (e.g., café -> cafe)

normalize_unicode

remove stop words

remove_stopword

tokenize words

tokenize_word

tokenize sentences

tokenize_sentence

substitute custom words (e.g., vs -> versus)

substitute_token

stem words

stem_word

lemmatize words

lemmatize_word

preprocess text through a sequence of preprocessing functions

preprocess_text

Release files for text-preprocessing 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for text-preprocessing 0.1.1
File Size Uploaded
text_preprocessing-0.1.1.tar.gz 13.9 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for text-preprocessing 0.1.1
File Interpreter ABI Platform
text_preprocessing-0.1.1-py2.py3-none-any.whl Python 2, Python 3 none any Details

Total release size: 23.5 kB

Release files / text_preprocessing-0.1.1.tar.gz

Download URL text_preprocessing-0.1.1.tar.gz
Size 13.9 kB
Tags Source
SHA-256 checksum
How to use checksums
81953b85c3ac6343a2013a9296191cadef8f14275867c7982a175f0d6d941b28
BLAKE2b-256 checksum
How to use checksums
71db92c1ce26b943e220819b094d8eade7122cefdc283e6fdf6699d798e9bf95
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.1 CPython/3.10.7

Release files / text_preprocessing-0.1.1-py2.py3-none-any.whl

Download URL text_preprocessing-0.1.1-py2.py3-none-any.whl
Size 9.6 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
848729c6f0dc8fd8d822e21744bc6eaf21ad01709bf438de62d37db4dbe2e61e
BLAKE2b-256 checksum
How to use checksums
8e15e2afbb516be264e15b2c82f2bb1dfbf902fd78e38e8b271fc7f3efe6df90
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.1 CPython/3.10.7

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files

0.1.0

2 release files

0.0.9

2 release files

0.0.8

2 release files

0.0.6

2 release files

0.0.4

2 release files

0.0.3

2 release files

0.0.2

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page