dobbi 0.10

An open-source NLP library: fast text cleaning and preprocessing.

Description

An open-source NLP library: fast text cleaning and preprocessing.

Overview

This library provides a quick and ready-to-use text preprocessing tools for text cleaning and normalization. You can simply remove hashtags, nicknames, emoji, url addresses, punctuation, whitespace and etc.

Installation

Getting it

To download dobbi, either fork this github repo or simply use Pypi via pip.

$ pip install dobbi

Usage

Import the library.

import dobbi

Interaction

The library uses method chaining in order to simplify text processing:

dobbi.clean()\
    .hashtag()\
    .nickname()\
    .url()\
    .execute('Check here: https://some-url.com')

Supported methods and patterns

The process consists of three stages:

1. Initialization methods: initialize a dobbi Work object
2. Intermediate methods: chain needed patterns in the needed order
3. Terminal methods:

Initialization functions:

• dobbi.clean()
• dobbi.collect()
• dobbi.replace()

Intermediate methods (pattern processing choice):

• regexp() - custom regular expressions
• url() - URLs
• html() - HTML and "<...>" type markups
• punctuation() - punctuation
• hashtag() - hashtags
• emoji() - emoji
• emoticons() - emoticons
• whitespace() - whitespaces
• nickname() - @-starting nicknames

Terminal methods:

• execute(str) - executes chosen methods on the provided string.
• function() - returns a function which is a combination of the chosen methods.

Examples

1. Clean a twitter message

dobbi.clean()\
    .hashtag()\
    .nickname()\
    .url()\
    .execute('#fun #lol    Why  @Alex33 is so funny? Check here: https://some-url.com')

Result: 'Why is so funny? Check here:'

2. Replace nickname and url with tokens

dobbi.replace()\
    .hashtag('')\
    .nickname()\
    .url('CUSTOM_URL_TOKEN')\
    .execute('#fun #lol    Why  @Alex33 is so funny? Check here: https://some-url.com')

Result: 'Why TOKEN_NICKNAME is so funny? Check here: CUSTOM_URL_TOKEN'

3. Get text cleanup function

func = dobbi.clean().url().hashtag().punctuation().whitespace().html().function()
func('\t #fun #lol    Why  @Alex33 is so... funny? <tag> \nCheck\there: https://some-url.com')

Result: 'Why Alex33 is so funny Check here'

(!) Please, try to avoid the in-line method chaining, as it is significantly less readable.

4. Chain regexp methods

dobbi.clean()\
    .regexp('#\w+')\
    .regexp('@\w+')\
    .regexp('https?://\S+')\
    .execute('#fun #lol    Why  @Alex33 is so funny? Check here: https://some-url.com')

Result: 'Why is so funny? Check here:'

Additional

Please pay attention that the functions are applied in the order you specify. So, you're better to chain .punctuation() as one of the last functions.