Efficient Numba based implementation of Gibbs Sampling Dirichlet Mixed Model.
Project description
tweetopic: Blazing Fast Topic modelling for Short Texts
:zap: Blazing Fast implementation of the Gibbs Sampling Dirichlet Mixture Model for topic modelling over short texts utilizing the power of :1234: Numpy and :snake: Numba.
The package uses the Movie Group Process algorithm described in Yin and Wang (2014).
Features
- Fast :zap:
- Scalable :collision:
- High consistency and coherence :dart:
- High quality topics :fire:
- Easy visualization and inspection :eyes:
- Full scikit-learn compatibility :nut_and_bolt:
🛠 Installation
Install from PyPI:
pip install tweetopic
If you intend to use the visualization features of PyLDAvis, install the package with optional dependencies:
pip install tweetopic[viz]
👩💻 Usage (documentation)
For easy topic modelling, tweetopic provides you the TopicPipeline class:
from tweetopic import TopicPipeline, DMM
from sklearn.feature_extraction.text import CountVectorizer
# Creating a vectorizer for extracting document-term matrix from the
# text corpus.
vectorizer = CountVectorizer(min_df=15, max_df=0.1)
# Creating a Dirichlet Multinomial Mixture Model with 30 components
dmm = DMM(n_clusters=30, n_iterations=100, alpha=0.1, beta=0.1)
# Creating topic pipeline
pipeline = TopicPipeline(vectorizer, dmm)
You may fit the model with a stream of short texts:
pipeline.fit(texts)
To examine the structure of the topics you can either look at the most frequently occuring words:
pipeline.top_words(top_n=3)
-----------------------------------------------------------------
[
{'vaccine': 1011.0, 'coronavirus': 428.0, 'vaccines': 396.0},
{'afghanistan': 586.0, 'taliban': 509.0, 'says': 464.0},
{'man': 362.0, 'prison': 310.0, 'year': 288.0},
{'police': 567.0, 'floyd': 444.0, 'trial': 393.0},
{'media': 331.0, 'twitter': 321.0, 'facebook': 306.0},
...
{'pandemic': 432.0, 'year': 427.0, 'new': 422.0},
{'election': 759.0, 'trump': 573.0, 'republican': 527.0},
{'women': 91.0, 'heard': 84.0, 'depp': 76.0}
]
Or use rich visualizations provided by pyLDAvis:
pipeline.visualize(texts)
Note: You must install optional dependencies if you intend to use pyLDAvis
🎓 References
- Yin, J., & Wang, J. (2014). A Dirichlet Multinomial Mixture Model-Based Approach for Short Text Clustering. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 233–242). Association for Computing Machinery.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for tweetopic-0.1.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5cec1b056fc3b4c9e09a0c6368a04a3ba5d931ded7cdc587ba1526afe2f38692 |
|
MD5 | 8abd945ea35278213a295bcfc580c5b0 |
|
BLAKE2b-256 | 37c17dc0a78da5929bd6bc2509af66a1068b9f5cf2dc3295ae709deba922e057 |