cleanmydata 0.1.0

A data cleaning library for text processing

cleanmydata

This library contains all the essential functions for data cleaning.

It takes a list of data cleaning parameters and either a string or pandas dataframe as input

Functions:

1. Remove new lines
2. Remove emails
3. Remove URLs
4. Remove hashtags (#hashtag)
5. Remove the string if it contains only numbers
6. Remove mentions (@user)
7. Remove retweets (RT...)
8. Remove text between the square brackets [ ]
9. Remove multiple whitespaces and replace with one whitespace
10. Replace characters with more than two occurrences and replace with one occurrence
11. Remove emojis
12. Count characters (only for dataframe; creates a new column)
13. Count words (only for dataframe; creates a new column)
14. Calculate average word length (only for dataframe; creates a new column)
15. Count stopwords (only for dataframe; creates two new columns, stowords and stopword_count)
16. Detect language (uses fasttext-langdetect) (only for dataframe; creates two new columns, lang and lang_prob)
17. Detect language (uses fasttext-langdetect) (only for dataframe; creates just one column with langauge and probability; takes less time)
18. Remove HTML tags

How to install?

pip install cleanmydata

Parameters

1. lst (list) - List of data cleaning operations
2. data (string or dataframe) - Data to be passed
3. column (string) - Dataframe column on which operation to perform; only for dataframe
4. save (boolean) - If you want to save the results in a new file
5. name (string) - Name of the new file if save is True

Usage

1. Import the library
   from cleanmydata.functions import *
2. Call the method clean_data, and pass the parameters as you wish.
3. By default, if the dataframe is passed, it drops all NA values (dropna)

Examples

1. To remove emails and hashtags
   mydata = "Hello folks. [e-mail] #hashtag"
mydata = clean_data(lst=[2, 4], data=mydata)
print(mydata)
2. To count stopwords, remove mentions, and URLs, and save file from a dataframe
   df = pd.read_csv('data/my_csv.csv', encoding='ISO-8859-1', dtype='unicode')
df = clean_data(lst=[15, 6, 2], data=df, column='comments', save=True, name='my custome file name')

Other notes

If using stopwords, make sure you have en_core_web_sm installed.
python -m spacy download en_core_web_sm

More options and enhancements coming soon...