Topic modeling for tweets and other short texts.
Project description
tweetopic: Blazing Fast Topic modelling for Short Texts
:zap: Blazing Fast topic modelling over short texts utilizing the power of :1234: Numpy and :snake: Numba.
Features
- Fast :zap:
- Scalable :collision:
- High consistency and coherence :dart:
- High quality topics :fire:
- Easy visualization and inspection :eyes:
- Full scikit-learn compatibility :nut_and_bolt:
🛠 Installation
Install from PyPI:
pip install tweetopic
If you intend to use the visualization features of PyLDAvis, install the package with optional dependencies:
pip install tweetopic[viz]
👩💻 Usage (documentation)
For easy topic modelling, tweetopic provides you the TopicPipeline class:
from tweetopic import TopicPipeline, DMM
from sklearn.feature_extraction.text import CountVectorizer
# Creating a vectorizer for extracting document-term matrix from the
# text corpus.
vectorizer = CountVectorizer(min_df=15, max_df=0.1)
# Creating a Dirichlet Multinomial Mixture Model with 30 components
dmm = DMM(n_clusters=30, n_iterations=100, alpha=0.1, beta=0.1)
# Creating topic pipeline
pipeline = TopicPipeline(vectorizer, dmm)
You may fit the model with a stream of short texts:
pipeline.fit(texts)
To examine the structure of the topics you can either look at the most frequently occuring words:
pipeline.top_words(top_n=3)
-----------------------------------------------------------------
[
{'vaccine': 1011.0, 'coronavirus': 428.0, 'vaccines': 396.0},
{'afghanistan': 586.0, 'taliban': 509.0, 'says': 464.0},
{'man': 362.0, 'prison': 310.0, 'year': 288.0},
{'police': 567.0, 'floyd': 444.0, 'trial': 393.0},
{'media': 331.0, 'twitter': 321.0, 'facebook': 306.0},
...
{'pandemic': 432.0, 'year': 427.0, 'new': 422.0},
{'election': 759.0, 'trump': 573.0, 'republican': 527.0},
{'women': 91.0, 'heard': 84.0, 'depp': 76.0}
]
Or use rich visualizations provided by pyLDAvis:
pipeline.visualize(texts)
Note: You must install optional dependencies if you intend to use pyLDAvis
🎓 References
- Yin, J., & Wang, J. (2014). A Dirichlet Multinomial Mixture Model-Based Approach for Short Text Clustering. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 233–242). Association for Computing Machinery.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
tweetopic-0.1.2.tar.gz
(14.7 kB
view hashes)
Built Distribution
Close
Hashes for tweetopic-0.1.2-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 39a3d00e83b6dd61ba210c33b0de1e15082a98ed6e38b61ed7057fc9bebadc2e |
|
MD5 | 10fe31f1758cc8852db0b05af9166be2 |
|
BLAKE2b-256 | 280d1c4dacceba7aadf347620d59882172be382e781c30f8031f560da5dac97c |