Skip to main content

textClust

textClust is a Python implementation of a stream-based text clustering framework. It processes text observations as they arrive, maintains evolving micro-clusters, periodically merges and fades clusters, and supports either token/TF-IDF representations or dense sentence embeddings. The repository contains online and offline clustering components, preprocessing utilities, stream loaders, history tracking, and runnable examples.

Installation

The project requires Python 3.11 or newer.

git clone https://github.com/lea11100/textClust.git
cd textClust
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e .

For the optional offline translation and embedding pipeline, install its extra dependencies:

python -m pip install -e ".[offline-preprocessing]"

This extra adds the OpenAI client, emoji utilities, and progress bars. Core sentence-transformer dependencies are already included in the main package.

Toy example

The following example creates an in-memory text stream and runs textClust with TF-IDF preprocessing. It does not require a dataset file or a downloaded embedding model.

from textClust.algorithms.presets import get_textClust
from textClust.data.point import RawPoint
from textClust.data.stream import Stream


class ToyStream(Stream):
    def load(self):
        texts = [
            "football team wins the match",
            "the team celebrates the football victory",
            "new python release improves package tools",
            "developers publish a new python library",
        ]
        for index, text in enumerate(texts):
            yield RawPoint(text=text, text_id=str(index), timestamp=index)
        yield None


clusterer = get_textClust(
    data_loader=ToyStream(),
    clean_interval=100,
    cluster_interval=10,
    log_interval=100,
    decay_rate=0.01,
    sigma=1,
)
clusterer.start()
print("Assigned clusters:", clusterer.history_manager.get_final_assignments())

Run it with:

python toy_example.py

The preset uses TFIDFPreprocessor, stopword removal, stemming, and fixed interval time management. nltk may need its English stopword resource before the first run:

import nltk
nltk.download("stopwords")

More complete examples are in examples/.

Preprocessing

Streaming TF-IDF preprocessing

TFIDFPreprocessor normalizes text and converts each observation into a term-frequency dictionary:

from textClust.preprocessing.preprocessor import TFIDFPreprocessor

preprocessor = TFIDFPreprocessor(
    stopword_removal=True,
    stemming=True,
    punctuation=True,
)

Streaming embedding preprocessing

EmbeddingPreprocessor encodes text with a local sentence_transformers.SentenceTransformer model:

from sentence_transformers import SentenceTransformer
from textClust.preprocessing.preprocessor import EmbeddingPreprocessor

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
preprocessor = EmbeddingPreprocessor(model, normalize=True)

The get_embClust preset performs this setup for you:

from textClust.algorithms.presets import get_embClust
from textClust.data.wrapper import StaticJsonWrapper

clusterer = get_embClust(
    data_loader=StaticJsonWrapper(
        "examples/data/raw_data/Semantic10-T.json"
    ),
    embedding_model_name="sentence-transformers/all-MiniLM-L6-v2",
)
clusterer.start()

OpenAI-backed offline preprocessing

The script src/textClust/preprocessing/offline_preprocessing.py prepares datasets for embedding:

  1. loads tweet IDs and text from configured .npy or .json files;
  2. converts emojis to readable text;
  3. translates non-English text to English through an OpenAI-compatible Responses API;
  4. writes a translation checkpoint once per dataset;
  5. encodes the translated text with configured Hugging Face sentence-transformer models;
  6. writes model-specific embedding checkpoints that can be reused by SimulatedEmbeddingPreprocessor.

Set the API key outside the source code:

export OPENAI_API_KEY="your-key"

The default base URL and model are configured for the Paderborn AI gateway, but they can be overridden:

export OPENAI_BASE_URL="https://your-openai-compatible-endpoint/v1"
export OPENAI_CHAT_MODEL="your-chat-model"

Do not commit credentials or pass API keys as command-line arguments. For HPC jobs, use the scheduler or cluster secret store. A user-only environment file outside the repository is another option:

chmod 600 ~/.config/textclust/openai.env
source ~/.config/textclust/openai.env

Run preprocessing for one or more configured datasets:

python src/textClust/preprocessing/offline_preprocessing.py \
  --datasets News-T Tweets-T

Use --datasets all (the default) to process every configured dataset. The script has a CUDA device setting at the top of the file; change DEVICE = "cuda" to DEVICE = "cpu" when CUDA is unavailable.

Dataset layout

Raw and preprocessed example data can be downloaded from Sciebo. Place downloaded datasets in examples/data/raw_data or in the corresponding preprocessed directories described below.

The pipeline uses this layout below examples/data:

examples/data/
├── raw_data/
│   └── <dataset_name>.npy or .json
├── translated_no_emojis/
│   └── <dataset_name>.npy
└── embedded/
    └── <model_name>/<dataset_name>.npy

For JSON input, each record should contain tweetID, tweets, and optionally ts/timestamp. .npy input is handled by SocialEDWrapper.

Using translated and embedded checkpoints

This package provides helper preprocessors that utilize precomputed translated or embedded tweets. The main function is to speed-up test runs or hyperparameter optimization. The inteded use is not for real-world applications.

To use a translation checkpoint with streaming TF-IDF clustering:

from textClust.preprocessing.preprocessor import (
    TFIDFPreprocessorWithTranslation,
)

preprocessor = TFIDFPreprocessorWithTranslation(
    dataset="News-T",
    folder="examples/data/translated_no_emojis",
)

To replay precomputed embeddings instead of encoding each tweet during runtime:

from textClust.preprocessing.preprocessor import (
    SimulatedEmbeddingPreprocessor,
)

preprocessor = SimulatedEmbeddingPreprocessor(
    model="sentence_transformers_all_MiniLM_L6_v2",
    dataset="News-T",
    folder="examples/data/embedded",
)

In both cases, pass the preprocessor to textClust as part of a custom pipeline and keep RawPoint.text_id unchanged. The offline script's model folder name and dataset name must match the model and dataset arguments used for the preprocessor.

Metadata

Release files for textclust 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for textclust 0.1.0
File Size Uploaded Provenance Actions
textclust-0.1.0.tar.gz 38.5 kB Without provenance Details for textclust-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for textclust 0.1.0
File Interpreter ABI Platform Actions
textclust-0.1.0-py3-none-any.whl Python 3 none any Details for textclust-0.1.0-py3-none-any.whl

Total release size: 82.9 kB

Release files / textclust-0.1.0.tar.gz

Download URL textclust-0.1.0.tar.gz
Size 38.5 kB
Tags Source
SHA-256 checksum
How to use checksums
7df75c7912b2c15b0a284e843e04ead00f05a7fe5c914cb5251b039f1cdadbd4
BLAKE2b-256 checksum
How to use checksums
ade1b58aeddbd148ac249a4ae17e68356dd5daf0a9ba6eef205e1a5849c26b48
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.5

Release files / textclust-0.1.0-py3-none-any.whl

Download URL textclust-0.1.0-py3-none-any.whl
Size 44.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
d8ac04985a7a8f13f9b82b43e6acee358981161fd599648efaecbb3abd4ac25e
BLAKE2b-256 checksum
How to use checksums
d58e6f02c747dae3d8dba426ab66e904dd549b2e57b2775e0ab7ad1e86d2658c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.5

Security for textclust 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page