Skip to main content

This code is updated by author to adapt to Python3. The former version of code implements a basic, Twitter-aware tokenizer. Originally developed by Christopher Potts and updated by the World Well-Being Project based out of the University of Pennsylvania and Stony Brook University. Shared with permission.

Project description

Happiest Fun Tokenizer for python3

This code is modified in order to adapt to Python3. And according to my own requirements, auther added some functions or params to preserve the specific tokens like nicknames, urls, and hashtags. The former version of code implements a basic, Twitter-aware tokenizer. Originally developed by Christopher Potts (Happy Fun Tokenizer) and updated by H. Andrew Schwartz. Shared with Christopher's permission. Further updated by Caspar Chan.

Usage

from happiestfuntokenizing.happiestfuntokenizing import Tokenizer

tokenizer = Tokenizer()

message = """OMG!!!! :) I looooooove this tokenizer lololol"""
tokens = tokenizer.tokenize(message)
print(tokens)
['omg', '!', '!', '!', '!', ':)', 'i', 'looooooove', 'this', 'tokenizer', 'lololol']

message = """OMG!!!! :) I looooooove this tokenizer LoLoLoLoLooOOOOL"""
tokenizer = Tokenizer(preserve_case=True)
tokens = tokenizer.tokenize(message)
print(tokens)
['OMG', '!', '!', '!', '!', ':)', 'I', 'looooooove', 'this', 'tokenizer', 'LoLoLoLoLooOOOOL']

message = """RT @cahnnle #happyfuncoding: this is a typical Twitter tweet :-) https://twiter/test.phd"""  
tokenizer = Tokenizer(preserve_keywords=True)  
tokens = tokenizer.tokenize(message)
print(tokens)
['rt', '@cahnnle', '#happyfuncoding', ':', 'this', 'is', 'a', 'typical', 'twitter', 'tweet', ':-)']   
print(tokenizer..get_preserve_dict())
{'username': ['@cahnnle'], 'url': ['https://twiter/test.phd'], 'hashtag': ['#happyfuncoding']}

Installation

This is available through pip

pip install happiestfuntokenizing

If you do not have sudo privileges you can use the --user flag

pip install --user happiestfuntokenizing

Requirements

This uses Python 3.* Package dependencies include re and html.

License

Licensed under a GNU General Public License v3 (GPLv3)

Background

Adapted by the World Well-Being Project based out of the University of Pennsylvania and Stony Brook University. Originally developed by Christopher Potts.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

happiestfuntokenizing-0.0.7.tar.gz (6.3 kB view details)

Uploaded Source

File details

Details for the file happiestfuntokenizing-0.0.7.tar.gz.

File metadata

  • Download URL: happiestfuntokenizing-0.0.7.tar.gz
  • Upload date:
  • Size: 6.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.5.0.1 requests/2.18.4 setuptools/40.6.3 requests-toolbelt/0.8.0 tqdm/4.26.0 CPython/3.6.3

File hashes

Hashes for happiestfuntokenizing-0.0.7.tar.gz
Algorithm Hash digest
SHA256 55289efd27faaa05b38cef9f6cb00ccc54e47c83addf6cc43e61ac6f86bdc213
MD5 ec21885d512edeca7e2e78186750aadb
BLAKE2b-256 38d06eeab22df5e414b456b3de8fdb71b866950b66ff9f8f19ea8af4dd8554af

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page