A simple tokenizer function for NLP
Project description
This tokenizer function turns text into lowercase word tokens, removes English stopwords, lemmatize the tokens and replaces URLs with a placeholder.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Close
Hashes for jf_tokenize_package-1.0.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 35e793637df1d1ed72d940dca19016896a67dbff06ce89f680457b5f6192ba1f |
|
MD5 | 9bd16f51bc7887f00064f46698a015d2 |
|
BLAKE2b-256 | 12c21c00049517b9ab0ddf8f96e2b3726e35ad0d72d0524efb2e90fc7a2e2e0c |