topicgpt 0.0.1.dev8

A package for integrating LLMs like GPT-3.5 and GPT-4 into topic modelling

TopicGPT

TopicGPT integrates the remarkable capabilities of current LLMs such as GPT-3.5 and GPT-4 into topic modelling.

While traditional topic models extract topics as simple lists of top-words, such as ["Lion", "Leopard", "Rhino", "Elephant", "Buffalo"], TopicGPT offers rich and dynamic topic representations that can be intuitively understood, extensively investigated and modified in various ways via a simple text commands.

More specifically, it provides the following core functionalities:

• Identification of clusters within document-embeddings and top-word extraction
• Generation of informative topic descriptions
• Extraction of detailed information about topics via Retrieval-Augmented-Generation (RAG)
• Comparison of topics
• Splitting and combining of identified topics
• Addition of new topics based on keywords
• Deletion of topics

It is further possible, to directly interact with TopicGPT via prompting and without explicitly calling functions - an LLM autonomously decides which functionality to use.

Installation

Example

The following example demonstrates how TopicGPT can be used on a real-world dataset. The Twenty Newsgroups corpus (https://scikit-learn.org/0.19/datasets/twenty_newsgroups.html) is used for this purpose.

Load the data

from sklearn.datasets import fetch_20newsgroups

data = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes')) #download the 20 Newsgroups dataset
corpus = data['data'] 

corpus = [doc for doc in corpus if doc != ""] #remove empty documents

Initialize the model

Note that an OpenAi API-Key is needed to compute the embeddings and execute the prompts. See https://platform.openai.com/account/api-keys for more details. We select 20 topics in this case since the Twenty Newsgroups corpus comprises documents from 20 different newsgroups. It is also possible to let Hdbscan determine the number of topics automatically.

from topicgpt.TopicGPT import TopicGPT

tm = TopicGPT(
    openai_api_key = <your-openai-api-key>,
    n_topics = 20 # select 20 topics since the true number of topics is 20 
)

Fit the model

The fit-method fits the model. This can take, depending on the size of the dataset and wether embeddings have been provided, from a few minutes to several hours. Especially the computation of the embeddings can take some time.

tm.fit(corpus) # the corpus argument should be of type list[str] where each string represents one document

Inspect the found topics

Obtain an overview over the indentified topics

print(tm.topic_lis)

Output

[Topic 0: Electronics Equipment Sales,
 Topic 1: Image Processing,
 Topic 2: Gun control,
 Topic 3: Online Privacy and Anonymity,
 Topic 4: Conflict and Violence.,
 Topic 5: Computer Hardware,
 Topic 6: Belief and Atheism,
 Topic 7: Online Discussions,
 Topic 8: Computer Software,
 Topic 9: Car Features and Performance,
 Topic 10: Encryption and Government,
 Topic 11: Technology and Computing.,
 Topic 12: Technology and Computing,
 Topic 13: Space Exploration,
 Topic 14: Motorcycle Riding Techniques,
 Topic 15: Technology,
 Topic 16: Hockey Games,
 Topic 17: Health and Medicine.,
 Topic 18: Baseball games and teams.,
 Topic 19: Beliefs about Homosexuality.]

To obtain more detailed information on each topic, we can call the "print_topics" method:

tm.print_topics()

Output

Topic 0: Electronics Equipment Sales

Topic_description: The common topic of the given words appears to be "electronics and technology". 

Various aspects and sub-topics of this topic include:
1. Buying and selling: "offer", "sale", "sell", "price", "buy"
2. Device usage and features: "use", "get", "new", "used", "condition"
3. Technical specifications: "wire", "ground", "power", "circuit", "voltage"
4. Communication and connectivity: "phone", "email", "modem", "wireless", "connection"
5. Accessories and peripherals: "battery", "cable", "manuals", "disk", "monitor"
Top words: ["n't", 'one', 'would', 'use', 'like', 'get', 'new', 'used', 'offer', 'sale']

[...]

We can also visualize the resulting clusters to get an overview of the shape and size of the clusters

tm.visualize_clusters()

Find out more detailed information about the identified topics

First, we might be interested in knowing what information the space topic (topic 13) contains on the moon landing.

tm.pprompt("Which information related to the keyword 'moon landing' does topic 13 have?")

Output

GPT wants to the call the function:  {
  "name": "knn_search",
  "arguments": "{\n  \"topic_index\": 13,\n  \"query\": \"moon landing\",\n  \"k\": 5\n}"
}
Topic 13, which is related to the keyword "moon landing," has the following information:

1. Document index 258: This document provides an introduction to the solar system and mentions that advancements in rocketry after World War II enabled machines to travel to the Moon and other planets. It highlights that the United States has sent both automated spacecraft and human-crewed expeditions to explore the Moon.

2. Document index 535: This document discusses a $65 million program called the Back to the Moon bill, which aims to encourage private companies to develop lunar orbiters. It mentions that there is a chance of making a lunar mission happen in this decade through this program.

3. Document index 357: This document is a request for more information on a recent newspaper article about the Japanese crashing or crash-landing a package on the Moon. It indicates that the article was vague and unclear.

4. Document index 321: This document speculates about what would have happened if the Soviets had beaten the United States in the Moon race. It suggests that the US would have still performed Moon landings and potentially set up a lunar base. The focus on Mars exploration would have depended on the Soviets' actions.

5. Document index 102: This document mentions the Hiten engineering-test mission, which spent time in a highly eccentric Earth orbit and performed lunar flybys before being inserted into lunar orbit using gravity-assist-like maneuvers. It states that the mission was expected to crash on the Moon eventually.

Please note that the above summaries are based on the content of the documents and may not capture all the information contained within them.

From this output we see that an instance of a GPT decided to call the function "knn_search" from the class "TopicPrompting". Indeed some documents on the topic "moon landing" have been found and the model summarizes the relevant information accordingly.

If we want to check, for instance the document with index 102 in topic 13 to learn more about the Hiten engineering-test mission, we can simply do the following:

print(tm.topic_lis[13].documents[535])

Output

Their Hiten engineering-test mission spent a while in a highly eccentric Earth orbit doing lunar flybys, and then was inserted into lunar orbit using some very tricky gravity-assist-like maneuvering.  This meant that it would crash on the Moon eventually, since there is no such thing as a stable lunar orbit (as far as anyone knows), and I believe I recall hearing recently that it was about to happen.

Split Topics

We find that topic 6 about Belief and Atheism is a bit general and would thus like to split see what 5 potential subtopics it contains:

tm.pprompt("What are 5 potential subtopics of topic 6")

Output

GPT wants to the call the function:  {
  "name": "split_topic_kmeans",
  "arguments": "{\n  \"topic_idx\": 6,\n  \"n_clusters\": 5\n}"
}

Here are five potential subtopics of topic 6:

1. Existence of God: This subtopic explores arguments, evidence, proofs, and the existence of God.

2. Atheism vs Theism: This subtopic delves into the concepts of atheism, theism, atheists, and theists, and the debates surrounding these belief systems.

3. Belief and Faith: This subtopic focuses on beliefs, faith, believers, and the roles they play in religious and philosophical contexts.

4. Rationality and Reasoning: This subtopic involves discussions on reason, rationality, logic, and argumentation as they relate to beliefs and debates about religion.

5. Morality and Ethics: This subtopic explores moral values, ethics, morals, and how they influence and are influenced by religious beliefs and debates.

These subtopics were inferred from the words provided and the descriptions of the topics in the corpus used for topic modeling.

Based on those results we decide to not actually split the topic into 5 subtopics but rather just 3 - One on "religious faith", one on "atheism" and one on "ethics and philisophy". We also want to actually make those changes to the existing topics, so we tell TopicGPT to do the changes inplace.

tm.pprompt("Please split topic 6 into subtopics based on the keywords 'religious faith', 'atheism' and 'ethics and philosophy'. Do this inplace.")

Output

I have split topic 6 into three subtopics based on the provided keywords: "religious faith", "atheism", and "ethics and philosophy". Here are the new subtopics:

1. Topic 21: Religious Beliefs
   - This topic encompasses the general theme of "Religion and Belief".
   - Subtopics and aspects of this topic include:
     - Faith and Beliefs: Believing, belief, true, evidence, truth.
     - Religious Practices: Praying, heretic, bible, church.
     - Religious Communities: Christian, Muslim, atheist.
     - Interpretation and Understanding: Interpretations, interpretation, understanding, context.
     - Morality and Ethics: Good, wrong, punishment, violence.

2. Topic 22: Existence of God
   - This topic focuses on "Atheism and Belief".
   - Subtopics and aspects of this topic include:
     - Arguments and Evidence: Atheists provide evidence for their beliefs.
     - Existence of God: Atheists question the existence of God.
     - Religious Beliefs: Atheists challenge religious beliefs.
     - Science and Reason: Atheists rely on science and reason.
     - Morality and Ethics: Atheists discuss morality and ethics.

3. Topic 23: Ethics and Morality
   - This topic revolves around "Morality and Ethics".
   - Subtopics and aspects of this topic include:
     - Moral reasoning: "Argument," "reasoning," "logic," "fallacy," "premises."
     - Ethical principles: "Right," "wrong," "immoral," "morals," "ethical."
     - Moral judgments: "Moral," "ethical," "morality," "judgments," "values."
     - Human behavior: "Homosexual," "sexual," "promiscuous," "behavior," "sex."
     - Scientific perspective: "Science," "scientific," "evidence," "empirical," "studies."
Please note that these subtopics are based on the keywords provided and may not cover all aspects of the original topic.

We see that TopicGPT performed the splitting as intended. However, the names and descriptions of the topics got changed slightly in order to optimally fit to the documents that actually constitute the topics based on the keywords. (If you like you can rename them at any time).

Combining topics

Since topics 15 ("Hockey Games") and 17 ("Baseball games and tems") are both about sports, we want to combine them into a single topic.

tm.pprompt("Please combine topics 15 and 17. Do this inplace.")

Output

GPT wants to the call the function:  {
  "name": "combine_topics",
  "arguments": "{\n  \"topic_idx_lis\": [15, 17],\n  \"inplace\": true\n}"
}
The topics 15 and 17 have been combined into a new topic called "Sports". This topic includes aspects and sub-topics related to sports such as team and players, games and seasons, performance and skills, fans and audience, and statistics and records. Some of the common words found in this topic include "team," "players," "hockey," "baseball," "game," "games," "season," "playoffs," "good," "better," "win," "hit," "score," "fans," "series," "watch," "fan," "stats," "record," "pts," and "career".

Tips and tricks for prompting TopicGPT

When using the "pprompt" or "prompt" function, TopicGPT can behave differently than intended. To alleviate those issues some simple tricks can help:

• Explizitly tell the model which function it should use and which paramters to select. (Sometimes the model simply cannot know what you except it to do.) For example, instead of using tm.pprompt("What are the subtopic of topic 13?"), use something like tm.pprompt("What are the subtopic of topic 13? Please use the function that uses the k-means algorithm to split the topic. Use a paramter of k = 5 and do this inplace")

• Just ask the same prompt again. Since TopicGPT is a stochastic system, calling the same function with the same argument again might yield a different functionality to be used or a different result.

• If this doesn't help, you can also directly call the function you want to use from the TopicPrompting class. In the example above you could do tm.topic_prompting.split_topic_kmeans(topic_idx = 13, n_clusters = 5, inplace = True). Note that all functions the model can call can also be called directly.

• In case of hallucination of facts it may help to use GPT-4 for TopicGPT

How TopicGPT works

Limitations and caveats

It is important to note that, as a model built on top of inherently stochastic LLMs and all their shortcomings, TopicGPT has several limitations and shortcomings as well. The following list is not aimed at being complete, but could provide useful information on what may go wrong when using TopicGPT:

• Hallucination: LLMs are well known for yielding incorrect but cohernt and plausible answers that seem convincing but are actually just made up. Although we tried to minimize this undesired behaviour through carefully designing the used prompts, we found that TopicGPT may hallucinate (especially) with respect to the following aspects:

  • Making up, distorting or misinterpreting content of documents retrieved via knn-search.
  • Incorrectly naming and describing topics based on top-words. Specifically, the model can identify topics that seem coherent and reasonable altough the corresponding documents are not actually related.

• Unsdesired Behaviour: When using the "prompt" or "pprompt" function, TopicGPT may not call the function you intended it to call. This can be alleviated by explizitly telling the model which function to use or directly calling the function yourself.

• Stoachasticity: The behaviour of TopicGPT is not deterministic and exhibits some randomness. There is alwyays some probability that certain actions do not work as intended at the first try because some components of the LLM do not function as desired. Simply trying again should mostly help with those issues.

  • On the other hand, TopicGPT may also be overly cautious and report that no relevant information has been found or no topic exists that matches a certain keyword even though it does. This could be caused by designing prompts to prevent massive occurence of falsely positive results. Note that using GPT-4 in TopicGPT can help to significantly alleviate issues with hallucination.

• Erroneous embeddings: The document- and word-embeddings used in TopicGPT may not always reflect the actual semantics of the texts correctly. More specifically, the embeddings sometimes reflect, for instance, grammatical or orthographical aspects such that clusters based on those aspects may be identified.

References

The follwing models, software packages and ideas are central for TopicGPT:

• UMAP: The Uniform Manifold Approximation and Projection for Dimension Reduction algorithm is used for reducing the dimensionality of document- and word embeddings (McInnes, Leland, John Healy, and James Melville. "Umap: Uniform manifold approximation and projection for dimension reduction." arXiv preprint arXiv:1802.03426 (2018).)
• HDBSCAN: Hierarchical density based clusterin is used to identify the clusters among the dimensionality reduced topics (McInnes, Leland, John Healy, and Steve Astels. "hdbscan: Hierarchical density based clustering." J. Open Source Softw. 2.11 (2017): 205.)
• Topword extraction: Even though the corresponding packages are not directly used, the topword extraction methods used for this package are based on very similar ideas as found in the Bertopic Model (Grootendorst, Maarten. "BERTopic: Neural topic modeling with a class-based TF-IDF procedure." arXiv preprint arXiv:2203.05794 (2022)) in the case of the tf-idf method and in Top2Vec for the centroid-similarity method (Angelov, Dimo. "Top2vec: Distributed representations of topics." arXiv preprint arXiv:2008.09470 (2020)).
• LLMs from the GPT family: Some references for the models for computing embeddings and answering the prompts include:
  • Brown, Tom B., et al. “Language Models are Few-Shot Learners.” Advances in Neural Information Processing Systems 33 (2020).
  • Radford, Alec, et al. “GPT-4: Generative Pre-training of Transformers with Discrete Latent Variables.” arXiv preprint arXiv:2302.07413 (2023).
  • Radford, Alec, et al. “Improving Language Understanding by Generative Pre-Training.” URL: https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf. [6]
  • Radford, Alec, et al. “Language Models are Unsupervised Multitask Learners.” OpenAI Blog 1.8 (2019): 9. [7]