dyf-rs 0.1.0

Fast outlier classification using PCA-based LSH (Rust core)

DYF - Outlier Classification

Fast outlier classification using PCA-based LSH. Identifies three types of items in embedding spaces:

• Dense: Items in well-populated semantic buckets
• Diaspora: Sparse items that find community via recovery PCA
• Orphan: Truly unique items with no semantic neighbors

Installation

pip install dyf

Quick Start

import numpy as np
from dyf import OutlierClassifier

# Create embeddings (e.g., from sentence-transformers)
embeddings = np.random.randn(10000, 384).astype(np.float32)

# Classify outliers
classifier = OutlierClassifier(embedding_dim=384)
classifier.fit(embeddings)

# Get results
print(classifier.report())
diaspora = classifier.get_diaspora()  # Indices of diaspora items
orphans = classifier.get_orphans()    # Indices of orphan items

Performance

~60ms for 60K embeddings (384 dimensions) - 3.8x faster than pure Python/sklearn.

API

OutlierClassifier

OutlierClassifier(
    embedding_dim: int,
    initial_bits: int = 14,      # Bits for initial PCA LSH
    recovery_bits: int = 8,       # Bits for recovery PCA
    dense_threshold: int = 10,    # Min bucket size for "dense"
    intra_outlier_std: float = 2.0,  # Std threshold for intra-bucket outliers
    recovery_cluster_min: int = 3,   # Min cluster size for "recovered"
    seed: int = 31
)

Methods:

• fit(embeddings) - Fit on numpy array (n_samples, embedding_dim)
• fit_arrow(arrow_array) - Fit on PyArrow FixedSizeListArray (zero-copy)
• get_diaspora() - Get indices of diaspora items
• get_orphans() - Get indices of orphan items
• get_statuses() - Get status for all items
• report() - Get classification report

License

MIT