Open Intent Classification

Closed intent classification uses a set of predefined labels to identify an intent. In comparison, open intent classification allows you to define as many labels you want, without fine tuning the model.

This project implements different components that support open intent classification such as an embedder, a T5 based fine tuned model for intent classification and a verbalizer. If you are interested in finer detailes you can read my blog post.

The goal of this library is to enable you test your assumptions about your data as fast as possible and to be a one stop shop for everything "classification like", similarly to how Bertopic is for clustering.

Why should you use this?

1. You are researching nlp classification problems and want to test different embeddings, verbalizers and components with plug-and-play feel
2. You want to detect semantically user intents in text but either don't want to commit to pre-defined classes OR just want to test out the fastest way to classify text other than through an LLM

[!IMPORTANT]

open-intent-classification project is in Alpha stage.

1. Expect API changes
2. Milage may vary. Quality of classifiers have been tested on Atis and Banking77

Installation

$ pip install open-intent-classifier

Usage

A full example is under Atis Notebook

T5 Based Intent Classification

from open_intent_classifier.model import IntentClassifier
model = IntentClassifier()
labels = ["Cancel Subscription", "Refund Requests", "Broken Item", "And More..."]
text = "I don't want to continue this subscription"
predicted_label = model.predict(text, labels)

By default, the IntentClassifier is loading a small model with 80M parameters.

For higher accuracy you can initialize the model with:

from open_intent_classifier.model import IntentClassifier
from open_intent_classifier.consts import INTENT_CLASSIFIER_248M_FLAN_T5_BASE
model = IntentClassifier(INTENT_CLASSIFIER_248M_FLAN_T5_BASE)

This will increase model latency as well.

Embeddings Based Classification

from open_intent_classifier.embedder import StaticLabelsEmbeddingClassifier
labels = ["Cancel Subscription", "Refund Requests", "Broken Item", "And More..."]
text = "I don't want to continue this subscription"
embeddings_classifier = StaticLabelsEmbeddingClassifier(labels)
predicted_label = embeddings_classifier.predict(text)

LLM Based Classification

Using LLM for classification is a viable option that sometimes provides the highest quality. Currently we have implemented only Open AI based LLMs.

from open_intent_classifier.model import OpenAiIntentClassifier
labels = ["Cancel Subscription", "Refund Requests", "Broken Item", "And More..."]
text = "I don't want to continue this subscription"
model_name = "gpt-4o-mini"
classifier = OpenAiIntentClassifier(model_name)
result = classifier.predict(text=text, labels=labels)

LLM Few Shot Based Classification

from open_intent_classifier.model import OpenAiIntentClassifier, ClassificationExample
labels = ["Cancel Subscription", "Refund Requests", "Broken Item", "And More..."]
text = "I don't want to continue this subscription"
model_name = "gpt-4o-mini"
example = ClassificationExample(text="I want to abort my account", intent_labels=labels, intent="Cancel Subscription") 
classifier = OpenAiIntentClassifier(model_name, few_shot_examples=[example])
result = classifier.predict(text=text, labels=labels)

DSPy based classification

Here's an example of using DSPy based classifier for both educational purposes and potential increase in quality.

from open_intent_classifier.model import DSPyClassifier
classifier = DSPyClassifier()
labels = ["Cancel subscription", "Refund request"]
text = "I want to cancel my subscription"
result = classifier.predict(text, labels)

SmolLM2 based classification

Here's an example of using SmolLM2 based classifier

from open_intent_classifier.model import SmolLm2Classifier
classifier = SmolLm2Classifier()
labels = ["Cancel subscription", "Refund request"]
text = "I want to cancel my subscription"
result = classifier.predict(text, labels)

SmolLM2 Few Shot Based Classification

from open_intent_classifier.model import SmolLm2Classifier

from open_intent_classifier.model import SmolLm2Classifier, ClassificationExample
labels = ["Cancel subscription", "Refund request"]
example = ClassificationExample(text="I want to abort my account", intent_labels=labels, intent="Cancel Subscription")
classifier = SmolLm2Classifier(few_shot_examples=[example])
text = "I want to cancel my subscription"
result = classifier.predict(text, labels)

Training the T5 base classifier

The details of training of the classifier is in another repository. I have separated training from inference in order to allow each repository to be focused and extended.

You can read about the training in the training repo: https://github.com/SerjSmor/intent_classification

Roadmap

• Add LLM based classification
• Add DSPy based classification
• Add small language models as classifiers
• Add few shot interface
• Add Atis and Banking77 comparisons with all classifiers
• Add embeddings filtering stage for classifiers
• Add multithreading for LLM based classifiers
• Add an option to ensemble embeddings and T5 (and additional models)
• Create a recommender for fine-tuning