Biterm Topic Model
Project description
Biterm Topic Model
Bitermplus implements Biterm topic model for short texts introduced by Xiaohui Yan, Jiafeng Guo, Yanyan Lan, and Xueqi Cheng. It uses Cython for better performance and includes two implementations of BTM: 1) bitermplus.BTM
class (based on BTM by Xiaohui Yan), 2) bitermplus.btm_depr.BTM
class (based on biterm package by Markus Tretzmüller). This package is also capable of computing perplexity and semantic coherence metrics.
Requirements
- Cython
- NumPy
- Pandas
- SciPy
- Scikit-learn
- pyLDAvis (optional)
Setup
You can install the package from PyPi:
pip install bitermplus
Or from this repo:
pip install git+https://github.com/maximtrp/bitermplus.git
Example
import bitermplus as btm
import numpy as np
from gzip import open as gzip_open
# Importing and vectorizing text data
with gzip_open('dataset/SearchSnippets.txt.gz', 'rb') as file:
texts = file.readlines()
# Vectorizing documents, obtaining full vocabulary and biterms
X, vocab = btm.util.get_words_freqs(texts)
docs_vec = btm.util.get_vectorized_docs(X)
biterms = btm.util.get_biterms(X)
# Initializing and running model
model = btm.BTM(X, T=8, W=vocab.size, M=20, alpha=50/8, beta=0.01)
model.fit(biterms, iterations=20)
p_zd = model.transform(docs_vec)
# Calculating metrics
perplexity = btm.metrics.perplexity(model.matrix_words_topics_, p_zd, X, 8)
coherence = btm.metrics.coherence(model.matrix_words_topics_, X, M=20)
# or
perplexity = model.perplexity_
coherence = model.coherence_
Acknowledgement
Markus Tretzmüller @markoarnauto
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
bitermplus-0.5.0.tar.gz
(724.7 kB
view hashes)