topicgpt-python 0.1.5

Official implementation of TopicGPT: A Prompt-based Topic Modeling Framework (NAACL'24)

TopicGPT

This repository contains scripts and prompts for our paper "TopicGPT: Topic Modeling by Prompting Large Language Models" (NAACL'24).

📣 Updates

• [11/09/24] Python package topicgpt_python is released! You can install it via pip install topicgpt_python. We support OpenAI API, Vertex AI, and vLLM (requires GPUs for inference).
• [11/18/23] Second-level topic generation code and refinement code are uploaded.
• [11/11/23] Basic pipeline is uploaded. Refinement and second-level topic generation code are coming soon.

📦 Using TopicGPT

Getting Started

1. Make a new Python 3.9+ environment using virtualenv or conda.
2. Install the required packages:

pip install topicgpt_python

• Set your API key:

export OPENAI_API_KEY={your_openai_api_key}
export VERTEX_PROJECT={your_vertex_project}
export VERTEX_LOCATION={your_vertex_location}
export HF_TOKEN={your_huggingface_token}

• Refer to https://openai.com/pricing/ for OpenAI API pricing or to https://cloud.google.com/vertex-ai/pricing for Vertex API pricing.

Data

• Prepare your .jsonl data file in the following format:

  {
      "id": "IDs (optional)",
      "text": "Documents",
      "label": "Ground-truth labels (optional)"
  }
  

• Put the data file in data/input. There is also a sample data file data/input/sample.jsonl to debug the code.
• #TODO: fix - If you want to sample a subset of the data for topic generation, run python script/data.py --data <data_file> --num_samples 1000 --output <output_file>. This will sample 1000 documents from the data file and save it to <output_file>. You can also specify --num_samples to sample a different number of documents, see the paper for more detail.
• Raw dataset used in the paper (Bills and Wiki): [link].

Pipeline

📜 Citation

@misc{pham2023topicgpt,
      title={TopicGPT: A Prompt-based Topic Modeling Framework}, 
      author={Chau Minh Pham and Alexander Hoyle and Simeng Sun and Mohit Iyyer},
      year={2023},
      eprint={2311.01449},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}