A python package for text preprocessing task in natural language processing.
Usage
To use this text preprocessing package, first install it using pip:
pip install text-preprocessing
Then, import the package in your python script and call appropriate functions:
from text_preprocessing import preprocess_text
from text_preprocessing import to_lower, remove_email, remove_url, remove_punctuation, lemmatize_word
# Preprocess text using default preprocess functions in the pipeline
text_to_process = 'Helllo, I am John Doe!!! My email is john.doe@email.com. Visit our website www.johndoe.com'
preprocessed_text = preprocess_text(text_to_process)
print(preprocessed_text)
# output: hello email visit website
# Preprocess text using custom preprocess functions in the pipeline
preprocess_functions = [to_lower, remove_email, remove_url, remove_punctuation, lemmatize_word]
preprocessed_text = preprocess_text(text_to_process, preprocess_functions)
print(preprocessed_text)
# output: helllo i am john doe my email is visit our website
Features
Feature |
Function |
|---|---|
convert to lower case |
to_lower |
convert to upper case |
to_upper |
keep only alphabetic and numerical characters |
keep_alpha_numeric |
check and correct spellings |
check_spelling |
expand contractions |
expand_contraction |
remove URLs |
remove_url |
remove names |
remove_name |
remove emails |
remove_email |
remove phone numbers |
remove_phone_number |
remove SSNs |
remove_ssn |
remove credit card numbers |
remove_credit_card_number |
remove numbers |
remove_number |
remove bullets and numbering |
remove_itemized_bullet_and_numbering |
remove special characters |
remove_special_character |
remove punctuations |
remove_punctuation |
remove extra whitespace |
remove_whitespace |
normalize unicode (e.g., café -> cafe) |
normalize_unicode |
remove stop words |
remove_stopword |
tokenize words |
tokenize_word |
tokenize sentences |
tokenize_sentence |
substitute custom words (e.g., vs -> versus) |
substitute_token |
stem words |
stem_word |
lemmatize words |
lemmatize_word |
preprocess text through a sequence of preprocessing functions |
preprocess_text |
Release files for text-preprocessing 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| text_preprocessing-0.1.1.tar.gz | 13.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| text_preprocessing-0.1.1-py2.py3-none-any.whl | Python 2, Python 3 | none | any | Details |
Total release size: 23.5 kB
Release files / text_preprocessing-0.1.1.tar.gz
| Download URL | text_preprocessing-0.1.1.tar.gz |
|---|---|
| Size | 13.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
81953b85c3ac6343a2013a9296191cadef8f14275867c7982a175f0d6d941b28
|
|
BLAKE2b-256 checksum How to use checksums |
71db92c1ce26b943e220819b094d8eade7122cefdc283e6fdf6699d798e9bf95
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.10.7
|
Release files / text_preprocessing-0.1.1-py2.py3-none-any.whl
| Download URL | text_preprocessing-0.1.1-py2.py3-none-any.whl |
|---|---|
| Size | 9.6 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
848729c6f0dc8fd8d822e21744bc6eaf21ad01709bf438de62d37db4dbe2e61e
|
|
BLAKE2b-256 checksum How to use checksums |
8e15e2afbb516be264e15b2c82f2bb1dfbf902fd78e38e8b271fc7f3efe6df90
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.10.7
|