hybrid-vectorizer 0.1.2

Unified embedding for tabular, text, and multimodal data

HybridVectorizer

Unified embedding for tabular, text, and multimodal data with powerful similarity search.

HybridVectorizer automatically handles mixed data types (numerical, categorical, text, dates) and creates high-quality vector representations for similarity search, recommendation systems, and machine learning pipelines.

HybridVectorizer

Late-fusion embeddings for mixed tabular + text data. One line to vectorize text + numeric + categorical into a single search space with adjustable block weights.

Quick links:
• PyPI · https://pypi.org/project/hybrid-vectorizer/
• Examples · https://github.com/hariharaprabhu/hybrid-vectorizer/tree/main/Examples
• Issues · https://github.com/hariharaprabhu/hybrid-vectorizer/issues

Quick Start

Basic Usage

import pandas as pd
from hybrid_vectorizer import HybridVectorizer

# Load financial data (S&P 500 companies)
df = pd.read_csv("sp500_companies.csv")

# Select relevant columns for analysis
df = df[['Symbol', 'Sector', 'Industry', 'Currentprice', 'Marketcap', 
         'Fulltimeemployees', 'Longbusinesssummary']]

# Initialize with company symbol as index
hv = HybridVectorizer(index_column="Symbol")
vectors = hv.fit_transform(df)

print(f"Generated {vectors.shape[0]} vectors with {vectors.shape[1]} features")

Finding Similar Companies

# Find companies similar to Google (GOOGL)
query = df.loc[df['Symbol']=='GOOGL'].iloc[0].to_dict()

results = hv.similarity_search(query, ignore_exact_matches=True, top_n=5)
print(results[['Symbol', 'Sector', 'Marketcap', 'similarity']])

Weight Tuning for Different Use Cases

Focus on Business Description (Text Similarity)

# Emphasize business model similarity
results = hv.similarity_search(
    query,
    block_weights={'text': 2.0, 'numerical': 0.5, 'categorical': 0.5},
    top_n=5
)
print("Companies with similar business models:")
print(results[['Symbol', 'Sector', 'Longbusinesssummary', 'similarity']])

Focus on Financial Metrics

# Emphasize financial similarity
results = hv.similarity_search(
    query,
    block_weights={'text': 0.3, 'numerical': 2.0, 'categorical': 0.5},
    top_n=5
)
print("Companies with similar financials:")
print(results[['Symbol', 'Currentprice', 'Marketcap', 'similarity']])

Focus on Industry/Sector

# Emphasize sector/industry similarity
results = hv.similarity_search(
    query,
    block_weights={'text': 0.3, 'numerical': 0.5, 'categorical': 2.0},
    top_n=5
)
print("Companies in similar sectors:")
print(results[['Symbol', 'Sector', 'Industry', 'similarity']])

Custom Queries

# Search for specific characteristics
custom_query = {
    'Sector': 'Technology',
    'Longbusinesssummary': 'cloud computing artificial intelligence',
    'Marketcap': 500000000000,  # $500B market cap
    'Fulltimeemployees': 100000
}

results = hv.similarity_search(custom_query, top_n=5)
print("Large tech companies with AI/cloud focus:")
print(results[['Symbol', 'Sector', 'Marketcap', 'similarity']])

Real-World Use Cases

• Investment Research: Find companies with similar business models or financials
• Competitor Analysis: Identify direct and indirect competitors
• Portfolio Construction: Build diversified portfolios based on similarity
• Market Research: Understand sector clustering and relationships

📦 Installation

pip install hybrid-vectorizer

GPU Support (Recommended for Better Performance)

For faster text embedding with large datasets, install GPU-accelerated PyTorch:

Check Your CUDA Version

nvidia-smi

Look for "CUDA Version: X.X" in the output.

Install GPU Support

# For CUDA 11.8 (most common)
pip install hybrid-vectorizer
pip install torch --index-url https://download.pytorch.org/whl/cu118

# For CUDA 12.1
pip install torch --index-url https://download.pytorch.org/whl/cu121

Verify GPU Installation

import torch
from hybrid_vectorizer import HybridVectorizer

print(f"CUDA available: {torch.cuda.is_available()}")
if torch.cuda.is_available():
    print(f"GPU: {torch.cuda.get_device_name()}")

# Test with your data
hv = HybridVectorizer()
# Should show "GPU detected" instead of "Using CPU"

Performance Notes

• CPU: Works well for datasets <1000 rows
• GPU: 5-10x faster for text embedding, recommended for larger datasets
• Text columns benefit most from GPU acceleration

Requirements:

• Python 3.8+
• pandas, numpy, scikit-learn
• sentence-transformers, torch

🏗️ Architecture

HybridVectorizer uses a novel late fusion approach for multimodal similarity search:

✨ Key Features

🔄 Automatic Data Type Handling

• Numerical: Auto-normalized with MinMaxScaler
• Categorical: One-hot or frequency encoding (smart threshold)
• Text: SentenceTransformer embeddings
• Dates: Extract features or ignore
• Mixed: Handles missing values, inf, NaN gracefully

🎯 Powerful Similarity Search

• Late Fusion: Combines modalities with configurable weights
• Block-level Control: Weight text vs. numerical vs. categorical separately
• Explanation: See which features drive similarity

🛠️ Production Ready

• Memory Efficient: Optimized for large datasets
• GPU Support: Automatic GPU detection for text encoding
• Persistence: Save/load trained models
• Error Handling: Informative custom exceptions

💡 Usage Examples

Basic Usage

# Fit and transform
hv = HybridVectorizer()
vectors = hv.fit_transform(df)

# Simple query
results = hv.similarity_search({'description': 'machine learning'})

Advanced Configuration

hv = HybridVectorizer(
    column_encodings={'description': 'text', 'category': 'categorical'},
    ignore_columns=['id', 'created_at'],
    index_column='id',
    onehot_threshold=15,
    text_batch_size=64
)

Weighted Search

# Emphasize text over numerical features
results = hv.similarity_search(
    query,
    block_weights={'text': 3, 'categorical': 2, 'numerical': 1}
)

Text-Only Search

results = hv.similarity_search(
    {'description': 'AI startup'}
)

🔧 Configuration Options

Parameter	Description	Default
column_encodings	Manual type overrides	{}
ignore_columns	Skip these columns	[]
index_column	ID column (preserved in results)	None
onehot_threshold	Max categories for one-hot encoding	10
default_text_model	SentenceTransformer model	'all-MiniLM-L6-v2'
text_batch_size	Batch size for text encoding	128

📊 Data Type Detection

HybridVectorizer automatically detects:

• Numerical: int64, float64, etc. → MinMax normalization
• Categorical: object with ≤10 unique values → One-hot encoding
• Text: object with >10 unique values → SentenceTransformer embeddings
• Dates: datetime64 → Extract year/month/day or ignore

Override with column_encodings={'col': 'text'} if needed.

🎛️ Advanced Features

Model Persistence

# Save trained model
hv.save('my_vectorizer.pkl')

# Load later
hv2 = HybridVectorizer.load('my_vectorizer.pkl')
results = hv2.similarity_search(query)

Encoding Report

# See how each column was processed
report = hv.get_encoding_report()
print(report)

External Vector Database

import faiss

# Use FAISS for faster search
index = faiss.IndexFlatIP(vectors.shape[1])
index.add(vectors)
hv.set_vector_db(index)

🚨 Error Handling

from hybrid_vectorizer import HybridVectorizerError, ModelNotFittedError

try:
    results = hv.similarity_search(query)
except ModelNotFittedError:
    print("Call fit_transform() first!")
except HybridVectorizerError as e:
    print(f"HybridVectorizer error: {e}")

📈 Performance

Typical performance on modern hardware:

Dataset Size	Fit Time	Search Time	Memory
1K rows	<1s	<1ms	~50MB
10K rows	<10s	<10ms	~200MB
100K rows	<2min	<100ms	~1GB

With mixed data types including text columns

🛠️ Development

# Clone repository
git clone https://github.com/hariharaprabhu/hybrid-vectorizer
cd hybrid-vectorizer

# Install in development mode
pip install -e .

# Run tests
python tests/test_basic.py

📄 License

Apache-2.0 License - see LICENSE file for details.

📞 Support

• Issues: GitHub Issues
• Documentation: See this README and docstrings
• Questions: Open an issue for questions or feature requests

---

HybridVectorizer - Making multimodal similarity search simple and powerful. 🚀