Contextualized Topic Models
Project description
Contextualized Topic Models
Contextualized Topic Models (CTM) are a family of topic models that use pre-trained representations of language (e.g., BERT) to support topic modeling. See the papers for details:
Cross-lingual Contextualized Topic Models with Zero-shot Learning https://arxiv.org/pdf/2004.07737v1.pdf
Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence https://arxiv.org/pdf/2004.03974.pdf
README
Make sure you read the doc a bit. The cross-lingual topic modeling requires to use a ZeroShot model and it is trained only on ONE language; with the power of multilingual BERT it can then be used to predict the topics of documents in unseen languages. For more details you can read the two papers mentioned above.
Jump start Tutorial
Name |
Link |
---|---|
CombinedTM for Wikipedia Documents |
|
CombinedTM with Preprocessing |
TL;DR
In CTMs we have two models. CombinedTM and ZeroShotTM, they have different use cases.
CTMs work better when the size of the bag of words has been restricted to a number of terms that does not go over 2000 elements (this is because we have a neural model that reconstructs the input bag of word). We have a preprocessing pipeline that can help you in dealing with this.
Check the BERT model you are using, the multilingual BERT model one used on English data might not give results that are as good as the pure English trained one.
Preprocessing is key. If you give BERT preprocessed text, it might be difficult to get out a good representation. What we usually do is use the preprocessed text for the bag of word creating and use the NOT preprocessed text for BERT embeddings. Our preprocessing class can take care of this for you.
ZeroShot Topic Model
Combined Topic Model
Software details:
Free software: MIT license
Documentation: https://contextualized-topic-models.readthedocs.io.
Super big shout-out to Stephen Carrow for creating the awesome https://github.com/estebandito22/PyTorchAVITM package from which we constructed the foundations of this package. We are happy to redistribute again this software under the MIT License.
Features
Combines BERT and Neural Variational Topic Models
Two different methodologies: Combined, where we combine BoW and BERT embeddings and ZeroShot, that uses only BERT embeddings
Includes methods to create embedded representations and BoW
Includes evaluation metrics
Overview
Important: If you want to use CUDA you need to install the correct version of the CUDA systems that matches your distribution, see pytorch.
Install the package using pip
pip install -U contextualized_topic_models
Contextual neural topic models can be easily instantiated using few parameters (although there is a wide range of parameters you can use to change the behaviour of the neural topic model). When you generate embeddings with BERT remember that there is a maximum length and for documents that are too long some words will be ignored.
An important aspect to take into account is which network you want to use: the one that combines BERT and the BoW or the one that just uses BERT. It’s easy to swap from one to the other:
ZeroShotTM:
ZeroShotTM(input_size=len(qt.vocab), bert_input_size=512, n_components=50)
CombinedTM:
CombinedTM(input_size=len(qt.vocab), bert_input_size=512, n_components=50)
But remember that you can do zero-shot cross-lingual topic modeling only with the ZeroShotTM
model. See cross-lingual-topic-modeling
Mono vs Multi-lingual Embeddings
All the examples below use a multilingual embedding model distiluse-base-multilingual-cased
.
If you are doing topic modeling in English, you can use the English sentence-bert model, bert-base-nli-mean-tokens. In that case,
it’s really easy to update the code to support mono-lingual English topic modeling.
qt = QuickText("bert-base-nli-mean-tokens",
text_for_bert=list_of_unpreprocessed_documents,
text_for_bow=list_of_preprocessed_documents)
In general, our package should be able to support all the models described in the sentence transformer package. and in HuggingFace.
Zero-Shot Cross-Lingual Topic Modeling
Our ZeroShotTM can be used for zero-shot topic modeling. It can handle words that are not used during the training phase. More interestingly, this model can be used for cross-lingual topic modeling! See the paper (https://arxiv.org/pdf/2004.07737v1.pdf)
The high level API to handle the text is pretty easy to use; text_for_bert should be used to pass to the model a list of documents that are not preprocessed. In this way, SBERT can use all the information in the text to generate the representations. Instead, to text_for_bow you should pass the pre-processed text used to build the BoW.
from contextualized_topic_models.models.ctm import ZeroShotTM
from contextualized_topic_models.utils.data_preparation import QuickText
from contextualized_topic_models.utils.data_preparation import bert_embeddings_from_file
from contextualized_topic_models.datasets.dataset import CTMDataset
qt = QuickText("distiluse-base-multilingual-cased",
text_for_bert=list_of_ENGLISH_unpreprocessed_documents,
text_for_bow=list_of_ENGLISH_preprocessed_documents)
training_dataset = qt.load_dataset()
ctm = ZeroShotTM(input_size=len(qt.vocab), bert_input_size=512, n_components=50)
ctm.fit(training_dataset) # run the model
ctm.get_topics()
Predict Topics for Unseen Documents
Once you have trained the cross-lingual topic model, you can use this simple pipeline to predict the topics for documents in a different language.
Note that the bag of words of the two languages will not be comparable!
qt = QuickText("distiluse-base-multilingual-cased",
text_for_bert=list_of_SPANISH_unpreprocessed_documents,
text_for_bow=list_of_SPANISH_preprocessed_documents)
testing_dataset = qt.load_dataset()
# n_sample how many times to sample the distribution (see the doc)
ctm.get_thetas(testing_dataset, n_samples=20)
Combined Topic Modeling
Here is how you can use the CombinedTM. Combined TM combines the BoW with SBERT, a process that seems to increase the coherence (https://arxiv.org/pdf/2004.03974.pdf).
from contextualized_topic_models.models.ctm import CombinedTM
from contextualized_topic_models.utils.data_preparation import QuickText
from contextualized_topic_models.utils.data_preparation import bert_embeddings_from_file
from contextualized_topic_models.datasets.dataset import CTMDataset
qt = QuickText("distiluse-base-multilingual-cased",
text_for_bert=list_of_unpreprocessed_documents,
text_for_bow=list_of_preprocessed_documents)
training_dataset = qt.load_dataset()
ctm = CombinedTM(input_size=len(qt.vocab), bert_input_size=512, n_components=50)
ctm.fit(training_dataset) # run the model
See the example notebook in the contextualized_topic_models/examples folder. We have also included some of the metrics normally used in the evaluation of topic models, for example you can compute the coherence of your topics using NPMI using our simple and high-level API.
from contextualized_topic_models.evaluation.measures import CoherenceNPMI
with open('preprocessed_documents.txt',"r") as fr:
texts = [doc.split() for doc in fr.read().splitlines()] # load text for NPMI
npmi = CoherenceNPMI(texts=texts, topics=ctm.get_topic_lists(10))
npmi.score()
Preprocessing
Do you need a quick script to run the preprocessing pipeline? we got you covered! Load your documents and then use our SimplePreprocessing class. It will automatically filter infrequent words and remove documents that are empty after training. The preprocess method will return the preprocessed and the unpreprocessed documents. We generally use the unpreprocessed for BERT and the preprocessed for the Bag Of Word.
from contextualized_topic_models.utils.preprocessing import WhiteSpacePreprocessing
documents = [line.strip() for line in open("unpreprocessed_documents.txt").readlines()]
sp = WhiteSpacePreprocessing(documents)
preprocessed_documents, unpreprocessed_documents, vocab = sp.preprocess()
Development Team
Federico Bianchi <f.bianchi@unibocconi.it> Bocconi University
Silvia Terragni <s.terragni4@campus.unimib.it> University of Milan-Bicocca
Dirk Hovy <dirk.hovy@unibocconi.it> Bocconi University
References
If you use this in a research work please cite these papers:
CombinedTM
@article{bianchi2020pretraining, title={Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence}, author={Federico Bianchi and Silvia Terragni and Dirk Hovy}, year={2020}, journal={arXiv preprint arXiv:2004.03974}, }
ZeroShotTM
@article{bianchi2020crosslingual, title={Cross-lingual Contextualized Topic Models with Zero-shot Learning}, author={Federico Bianchi and Silvia Terragni and Dirk Hovy and Debora Nozza and Elisabetta Fersini}, year={2020}, journal={arXiv preprint arXiv:2004.07737}, }
Credits
This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template. To ease the use of the library we have also included the rbo package, all the rights reserved to the author of that package.
Note
Remember that this is a research tool :)
History
1.6.1 (2020-12-10)
adding a new and much simpler way to handle text for topic modeling
1.6.0 (2020-11-03)
introducing the two different classes for ZeroShotTM and CombinedTM
depracating CTM class in favor of ZeroShotTM and CombinedTM
1.5.3 (2020-11-03)
adding support for Windows encoding by defaulting file load to UTF-8
1.5.2 (2020-11-03)
updated sentence-transformers version to 0.3.6
beta support for model saving and loading
new evaluation metrics based on coherence
1.5.0 (2020-09-14)
Introduced a method to predict the topics for a set of documents (supports multiple sampling to reduce variation)
Adding some features to bert embeddings creation like increased batch size and progress bar
Supporting training directly from lists without the need to deal with files
Adding a simple quick preprocessing pipeline
1.4.3 (2020-09-03)
Updating sentence-transformers package to avoid errors
1.4.2 (2020-08-04)
Changed the encoding on file load for the SBERT embedding function
1.4.1 (2020-08-04)
Fixed bug over sparse matrices
1.4.0 (2020-08-01)
New feature handling sparse bow for optimized processing
New method to return topic distributions for words
1.0.0 (2020-04-05)
Released models with the main features implemented
0.1.0 (2020-04-04)
First release on PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for contextualized_topic_models-1.7.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 77b330fb8568376a787baa2706bc2a7bc31a443889fa5990e400dbbff8d2182d |
|
MD5 | d4930c83c3f2394200ced03075829995 |
|
BLAKE2b-256 | 52e7ae7c9d24432741656807f97b870ed8774a75a09ce5f58f4b1dd6b1efbbdd |
Hashes for contextualized_topic_models-1.7.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7cf3ce1d6d892c89cb6123c192822dc65090d22a5850f56d315ce61061fc05df |
|
MD5 | cf2e6d944c8ab6edbf1260c7ec4b7e1e |
|
BLAKE2b-256 | ca686d32f226378e5f149767c14950c3b7da3e972be4ba6dd2121348d2943105 |