This code is updated by author to adapt to Python3. The former version of code implements a basic, Twitter-aware tokenizer. Originally developed by Christopher Potts and updated by the World Well-Being Project based out of the University of Pennsylvania and Stony Brook University. Shared with permission.
Project description
Happiest Fun Tokenizer for python3
This code is modified in order to adapt to Python3. And according to my own requirements, auther added some functions or params to preserve the specific tokens like nicknames, urls, and hashtags. The former version of code implements a basic, Twitter-aware tokenizer. Originally developed by Christopher Potts (Happy Fun Tokenizer) and updated by H. Andrew Schwartz. Shared with Christopher's permission. Further updated by Caspar Chan.
Usage
from happiestfuntokenizing.happiestfuntokenizing import Tokenizer
tokenizer = Tokenizer()
message = """OMG!!!! :) I looooooove this tokenizer lololol"""
tokens = tokenizer.tokenize(message)
print(tokens)
['omg', '!', '!', '!', '!', ':)', 'i', 'looooooove', 'this', 'tokenizer', 'lololol']
message = """OMG!!!! :) I looooooove this tokenizer LoLoLoLoLooOOOOL"""
tokenizer = Tokenizer(preserve_case=True)
tokens = tokenizer.tokenize(message)
print(tokens)
['OMG', '!', '!', '!', '!', ':)', 'I', 'looooooove', 'this', 'tokenizer', 'LoLoLoLoLooOOOOL']
message = """RT @cahnnle #happyfuncoding: this is a typical Twitter tweet :-) https://twiter/test.phd"""
tokenizer = Tokenizer(preserve_keywords=True)
tokens = tokenizer.tokenize(message)
print(tokens)
['rt', '@cahnnle', '#happyfuncoding', ':', 'this', 'is', 'a', 'typical', 'twitter', 'tweet', ':-)']
print(tokenizer..get_preserve_dict())
{'username': ['@cahnnle'], 'url': ['https://twiter/test.phd'], 'hashtag': ['#happyfuncoding']}
Installation
This is available through pip
pip install happiestfuntokenizing
If you do not have sudo privileges you can use the --user
flag
pip install --user happiestfuntokenizing
Requirements
This uses Python 3.* Package dependencies include re
and html
.
License
Licensed under a GNU General Public License v3 (GPLv3)
Background
Adapted by the World Well-Being Project based out of the University of Pennsylvania and Stony Brook University. Originally developed by Christopher Potts.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for happiestfuntokenizing-0.0.7.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 55289efd27faaa05b38cef9f6cb00ccc54e47c83addf6cc43e61ac6f86bdc213 |
|
MD5 | ec21885d512edeca7e2e78186750aadb |
|
BLAKE2b-256 | 38d06eeab22df5e414b456b3de8fdb71b866950b66ff9f8f19ea8af4dd8554af |