slenps 0.1.2

Some Pipelines

slenps

slenps is a collection of some simple NLP pipelines

• ecluster: cluster word embeddings
• llmclf: binary and multiclass classification using LLM

Installation

PyPI

pip install slenps

Example usage

embed documents and cluster its embeddings

Cluster some word with chosen embedding and clustering models

from slenps.eclusters import load_embedding_model, get_clustering_model_dict, load_clustering_model, cluster
import numpy as np 

# load documents

with open('sample_documents.txt', 'r') as file:
    documents = np.array([line.strip() for line in file.readlines()])

# embedding model
embedding_model = load_embedding_model(
    model_name = 'all-MiniLM-L6-v2', mode = 'huggingface',
) 
# embedding_model = load_embedding_model(model_name='word2vec') # or 'tfidf', 'doc2vec'

# obtain embeddings 
embeddings = embedding_model.encode(documents)

# clustering model
clustering_model = load_clustering_model('kmeans')
clustering_model = clustering_model.set_params(n_clusters=3)

# fit the model and retrieve labels and metrics
labels, metrics = cluster(
    embeddings, clustering_model, 
    metrics = ['dbs', 'calinski'],
)
print(metrics)

# print sample result
n_samples = 10
for document, label in zip(documents[:n_samples], labels[:n_samples]):
    print(f'{document} -> label {label}')

Find the best algorithm and num_cluster

from slenps.eclusters import find_best_algorithm
import pandas as pd
# define a list of clustering models to evaluate
# all default models are included in get_clustering_model_dict
model_names = ['kmeans', 'agglomerative_clustering', 'spectral_clustering']

# find best algo and num_cluster using test_metric
results = find_best_algorithm(
	embeddings, model_names=model_names,
	test_metric='dbs', metrics = ['dbs', 'silhouette'],
	min_cluster_num=2, max_cluster_num=10,
	result_filepath='sample_result_metrics.csv',
	print_topk=True,
)

# view all results
print(pd.DataFrame(results))

Supported models

Embedding models

embedding model	model_name	mode
sklearn.feature.extraction.text.TfidfVectorizer	tfidf	None
gensim.models.Word2Vec	word2vec	None
gensim.models.Doc2Vec	doc2vec	None
sentence_transformers.SentenceTransformer	Any	huggingface

Clustering models

clustering model	model_name	default params
sklearn.cluster.KMeans	kmeans	n_init='auto'
sklearn.cluster.AgglomerativeClustering	agglomerative_clustering	None
sklearn.cluster.SpectralClustering	spectral_clustering	None
sklearn.cluster.MeanShift	mean_shift	None
sklearn.cluster.AffinityPropagation	affinity_propagation	None
sklearn.cluster.Birch	birch	threshold=0.2