An open-source python package to clean raw text data
Project description
cleantext
cleantext is a an open-source python package to clean raw text data. Source code for the library can be found here.
Features
cleantext has two main methods,
- clean: to clean raw text and return the cleaned text
- clean_words: to clean raw text and return a list of clean words
cleantext can apply all, or a selected combination of the following cleaning operations:
- Remove extra white spaces
- Convert the entire text into a uniform lowercase
- Remove digits from the text
- Remove punctuations from the text
- Remove stop words, and choose a language for stop words ( Stop words are generally the most common words in a language with no significant meaning such as is, am, the, this, are etc.)
- Stem the words (Stemming is a process of converting words with similar meaning into a single word. For example, stemming of words run, runs, running will result run, run, run)
Installation
cleantext requires Python 3 and NLTK to execute.
To install using pip, use
pip install cleantext
Usage
- Import the library:
import cleantext
- Choose a method:
To return the text in a string format,
cleantext.clean("your_raw_text_here", all= True)
To return a list of words from the text,
cleantext.clean_words("your_raw_text_here", all= True)
To choose a specific set of cleaning operations,
cleantext.clean_words("your_raw_text_here",
all= False # Execute all cleaning operations
extra_spaces=True , # Remove extra white space
stemming=True , # Stem the words
stopwords=True ,# Remove stop words
lowercase=True ,# Convert to lowercase
numbers=True ,# Remove all digits
punct=True ,# Remove all punctuations
stp_lang='english' # Language for stop words
)
Examples
import cleantext
cleantext.clean('This is A s$ample !!!! tExt3% to cleaN566556+2+59*/133', extra_spaces=True, lowercase=True, numbers=True, punct=True)
returns,
'this is a sample text to clean'
import cleantext
cleantext.clean_words('This is A s$ample !!!! tExt3% to cleaN566556+2+59*/133', all=True)
returns,
['sampl', 'text', 'clean']
License
MIT
For any questions, issues, bugs, and suggestions please visit here
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
cleantext-1.1.3.tar.gz
(2.6 kB
view hashes)
Built Distribution
Close
Hashes for cleantext-1.1.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 82fd5525537bb3686778d7703ee1ebc0d07f0845057a14ee9d2a5708e8ef5192 |
|
MD5 | 34bf1bd4786e6f3dd94917308c9acdcd |
|
BLAKE2b-256 | 4948897230ef3a3544e5fab107b2e999729ea46091b6eabdcb95c3e5760dbd65 |