textClust
textClust is a Python implementation of a stream-based text clustering framework. It processes text observations as they arrive, maintains evolving micro-clusters, periodically merges and fades clusters, and supports either token/TF-IDF representations or dense sentence embeddings. The repository contains online and offline clustering components, preprocessing utilities, stream loaders, history tracking, and runnable examples.
Installation
The project requires Python 3.11 or newer.
git clone https://github.com/lea11100/textClust.git
cd textClust
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e .
For the optional offline translation and embedding pipeline, install its extra dependencies:
python -m pip install -e ".[offline-preprocessing]"
This extra adds the OpenAI client, emoji utilities, and progress bars. Core sentence-transformer dependencies are already included in the main package.
Toy example
The following example creates an in-memory text stream and runs textClust with TF-IDF preprocessing. It does not require a dataset file or a downloaded embedding model.
from textClust.algorithms.presets import get_textClust
from textClust.data.point import RawPoint
from textClust.data.stream import Stream
class ToyStream(Stream):
def load(self):
texts = [
"football team wins the match",
"the team celebrates the football victory",
"new python release improves package tools",
"developers publish a new python library",
]
for index, text in enumerate(texts):
yield RawPoint(text=text, text_id=str(index), timestamp=index)
yield None
clusterer = get_textClust(
data_loader=ToyStream(),
clean_interval=100,
cluster_interval=10,
log_interval=100,
decay_rate=0.01,
sigma=1,
)
clusterer.start()
print("Assigned clusters:", clusterer.history_manager.get_final_assignments())
Run it with:
python toy_example.py
The preset uses TFIDFPreprocessor, stopword removal, stemming, and fixed
interval time management. nltk may need its English stopword resource before
the first run:
import nltk
nltk.download("stopwords")
More complete examples are in examples/.
Preprocessing
Streaming TF-IDF preprocessing
TFIDFPreprocessor normalizes text and converts each observation into a
term-frequency dictionary:
from textClust.preprocessing.preprocessor import TFIDFPreprocessor
preprocessor = TFIDFPreprocessor(
stopword_removal=True,
stemming=True,
punctuation=True,
)
Streaming embedding preprocessing
EmbeddingPreprocessor encodes text with a local
sentence_transformers.SentenceTransformer model:
from sentence_transformers import SentenceTransformer
from textClust.preprocessing.preprocessor import EmbeddingPreprocessor
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
preprocessor = EmbeddingPreprocessor(model, normalize=True)
The get_embClust preset performs this setup for you:
from textClust.algorithms.presets import get_embClust
from textClust.data.wrapper import StaticJsonWrapper
clusterer = get_embClust(
data_loader=StaticJsonWrapper(
"examples/data/raw_data/Semantic10-T.json"
),
embedding_model_name="sentence-transformers/all-MiniLM-L6-v2",
)
clusterer.start()
OpenAI-backed offline preprocessing
The script
src/textClust/preprocessing/offline_preprocessing.py
prepares datasets for embedding:
- loads tweet IDs and text from configured
.npyor.jsonfiles; - converts emojis to readable text;
- translates non-English text to English through an OpenAI-compatible Responses API;
- writes a translation checkpoint once per dataset;
- encodes the translated text with configured Hugging Face sentence-transformer models;
- writes model-specific embedding checkpoints that can be reused by
SimulatedEmbeddingPreprocessor.
Set the API key outside the source code:
export OPENAI_API_KEY="your-key"
The default base URL and model are configured for the Paderborn AI gateway, but they can be overridden:
export OPENAI_BASE_URL="https://your-openai-compatible-endpoint/v1"
export OPENAI_CHAT_MODEL="your-chat-model"
Do not commit credentials or pass API keys as command-line arguments. For HPC jobs, use the scheduler or cluster secret store. A user-only environment file outside the repository is another option:
chmod 600 ~/.config/textclust/openai.env
source ~/.config/textclust/openai.env
Run preprocessing for one or more configured datasets:
python src/textClust/preprocessing/offline_preprocessing.py \
--datasets News-T Tweets-T
Use --datasets all (the default) to process every configured dataset. The
script has a CUDA device setting at the top of the file; change
DEVICE = "cuda" to DEVICE = "cpu" when CUDA is unavailable.
Dataset layout
Raw and preprocessed example data can be downloaded from
Sciebo. Place downloaded
datasets in examples/data/raw_data or in the corresponding preprocessed
directories described below.
The pipeline uses this layout below examples/data:
examples/data/
├── raw_data/
│ └── <dataset_name>.npy or .json
├── translated_no_emojis/
│ └── <dataset_name>.npy
└── embedded/
└── <model_name>/<dataset_name>.npy
For JSON input, each record should contain tweetID, tweets, and
optionally ts/timestamp. .npy input is handled by SocialEDWrapper.
Using translated and embedded checkpoints
This package provides helper preprocessors that utilize precomputed translated or embedded tweets. The main function is to speed-up test runs or hyperparameter optimization. The inteded use is not for real-world applications.
To use a translation checkpoint with streaming TF-IDF clustering:
from textClust.preprocessing.preprocessor import (
TFIDFPreprocessorWithTranslation,
)
preprocessor = TFIDFPreprocessorWithTranslation(
dataset="News-T",
folder="examples/data/translated_no_emojis",
)
To replay precomputed embeddings instead of encoding each tweet during runtime:
from textClust.preprocessing.preprocessor import (
SimulatedEmbeddingPreprocessor,
)
preprocessor = SimulatedEmbeddingPreprocessor(
model="sentence_transformers_all_MiniLM_L6_v2",
dataset="News-T",
folder="examples/data/embedded",
)
In both cases, pass the preprocessor to textClust as part of a custom
pipeline and keep RawPoint.text_id unchanged. The offline script's model
folder name and dataset name must match the model and dataset arguments
used for the preprocessor.
Metadata
Release files for textclust 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| textclust-0.1.0.tar.gz | 38.5 kB | Without provenance | Details for textclust-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| textclust-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for textclust-0.1.0-py3-none-any.whl |
Total release size: 82.9 kB
Release files / textclust-0.1.0.tar.gz
| Download URL | textclust-0.1.0.tar.gz |
|---|---|
| Size | 38.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7df75c7912b2c15b0a284e843e04ead00f05a7fe5c914cb5251b039f1cdadbd4
|
|
BLAKE2b-256 checksum How to use checksums |
ade1b58aeddbd148ac249a4ae17e68356dd5daf0a9ba6eef205e1a5849c26b48
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.5
|
Release files / textclust-0.1.0-py3-none-any.whl
| Download URL | textclust-0.1.0-py3-none-any.whl |
|---|---|
| Size | 44.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d8ac04985a7a8f13f9b82b43e6acee358981161fd599648efaecbb3abd4ac25e
|
|
BLAKE2b-256 checksum How to use checksums |
d58e6f02c747dae3d8dba426ab66e904dd549b2e57b2775e0ab7ad1e86d2658c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.5
|
Security for textclust 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.