Statistics for each tag's set of key phrases
Project description
A concise yet efficient implementation for computing the statistics of each tag's set of key phrases over input lines in Python 3. One of the major applications is for sentiment analysis, where each tag is a sentiment with the respective key phrases describing the sentiment.
How it Works
A trie structure is constructed to index all the key phrases. Then each line is matched towards the index to compute the respective statistics. The time complexity is $O(m^2 \cdot n)$, where $m$ is the maximum number of words in each line and $n$ is the number of lines.
Installation
This package is available on PyPI. Just use pip3 install -U TagStats
to install it.
Usage
You can simply call compute(content, tagDict)
, where content
is a list of lines and tagDict
is a dictionary with each tag name as key and the respective set of key phrases as value.
from tagstats import compute
print(compute(
[
"a b c",
"b c",
"a c e"
],
{
"+": ["a b", "a c"],
"-": ["b c"]
}
))
The output is a dictionary with each tag name as key and the respective totaled frequencies for lines as value.
{'+': [1, 0, 1], '-': [1, 1, 0]}
You can change the default tokenizer, by specifying a compiled regex as separator to tagstats.tagstats.tokenizer
. You can disable the tokenizer to allow matching over word boundaries, by specifying None
.
You can pre-build the index by calling index(tagDict)
, and later reuse it more than once as an optional parameter of compute(content, tagDict, index)
.
Tip
I strongly encourage using PyPy instead of CPython to run the script for best performance.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.