perceptionml 0.1.0

A text embedding analysis pipeline for perception modeling

perceptionML

A text embedding analysis pipeline for perception modeling and topic discovery.

Features

• Generate text embeddings using state-of-the-art models (Sentence Transformers, OpenAI, etc.)
• Dimensionality reduction with UMAP or PCA
• Advanced clustering with HDBSCAN
• Interactive HTML visualizations
• Topic analysis and statistics
• Support for zero-presence analysis and category comparisons
• Multi-GPU support for large datasets

Installation

pip install perceptionml

Quick Start

Simplest Usage - No Configuration Needed!

Just point to your CSV file with text:

perceptionml --data your_data.csv

perceptionML will automatically:

• Detect your text column (longest text)
• Find or create an ID column
• Identify numeric columns as outcomes
• Generate synthetic outcomes if no numeric columns exist
• Use optimal settings for finding many detailed topics

What Your Data Should Look Like

Minimal CSV (just text):

text
"This is my first document about..."
"Another document with different content..."

CSV with outcomes to analyze:

id,text,sentiment_score,rating
1,"Great product, highly recommend!",0.95,5
2,"Terrible experience, would not buy again",-0.87,1

Basic Options

# Specify output file name
perceptionml --data your_data.csv --output my_analysis.html

# Sample large datasets
perceptionml --data your_data.csv --sample-size 10000

# Use specific embedding model
perceptionml --data your_data.csv --embedding-model nvidia/NV-Embed-v2

# Export results to CSV
perceptionml --data your_data.csv --export-csv

Advanced Usage

For more control, you can:

1. Use configuration files for complex setups

2. Adjust clustering granularity:

   # Many small topics (default)
   perceptionml --data your_data.csv --auto-cluster many
   
   # Medium-sized topics  
   perceptionml --data your_data.csv --auto-cluster medium
   
   # Few large topics
   perceptionml --data your_data.csv --auto-cluster few
   

3. Override specific parameters:

   perceptionml --data your_data.csv \
       --min-cluster-size 30 \
       --umap-neighbors 15
   

Understanding the Output

The HTML visualization shows:

• 3D scatter plot of your texts, clustered by topic
• Topic keywords extracted from each cluster
• Statistics about outcomes in different regions
• Interactive controls to explore the data

Click on points to read the original texts. Use the controls to filter by outcome values or focus on specific topics.

Requirements

• Python 3.8+
• CUDA-capable GPU recommended for faster embedding generation
• 4GB+ RAM for typical datasets

Support

• Documentation: https://github.com/raymondli/perceptionml
• Issues: https://github.com/raymondli/perceptionml/issues
• Author: Raymond V. Li ([e-mail])

License

See LICENSE file for details.