Surrogate Concept Retrieval

surrogate_concept_retrieval

Implementation for the paper Concept Retrieval - What and How?

Package Status

✅ Added:

• Project URLs and Documentation links
• Keywords and classifiers for PyPI
• Populated __init__.py for proper imports
• Documentation structure with Sphinx
• Example code
• Improved README with usage examples

🔄 In Progress:

• Comprehensive documentation
• Test coverage
• CI/CD setup

Getting Started

# Install the package
pip install -e .

See RECOMMENDATIONS.md for full details on package improvements.

Overview

This package provides tools for extracting concepts from large datasets using surrogate concept retrieval method.

Features

• Fast embedding indexing using FAISS
• GPU-accelerated similarity computation
• Automatic concept extraction from embedding spaces
• Flexible concept filtering and refinement
• Support for projection-based concept analysis

Installation

# Install from PyPI
pip install coret

# Install with development dependencies
pip install "coret[dev]"

Quick Start

import numpy as np
from coret import ConceptRetrieval

# Load your embeddings (example uses random data)
embeddings = np.random.randn(1000, 768)
embeddings = embeddings / np.linalg.norm(embeddings, axis=1, keepdims=True)
embeddings = np.ascontiguousarray(embeddings, dtype=np.float32)

# Initialize concept retrieval
concept_retriever = ConceptRetrieval()

# Fit the model with embeddings
concept_retriever.fit(embeddings=embeddings)

# Select a random query embedding for demonstration
query_index = np.random.randint(0, len(embeddings))
query_embedding = embeddings[query_index]

# Retrieve concepts for the query
concepts = concept_retriever.retrieve(
    query=query_embedding,
    number_of_concepts=5,
    number_of_samples_per_concept=5
)

# Print retrieved concepts
top_k_concepts_indices_s = concepts['top_k_concepts_indices_s']

print(f"Query index: {query_index}")
for i, concept_indices in enumerate(top_k_concepts_indices_s):
  print(f"Concept {i+1}:")
  print(f"  Indices: {concept_indices}")
  print()

Requirements

• Python 3.9+
• CUDA-compatible GPU (recommended for large datasets)
• Dependencies:
  • numpy
  • faiss-gpu (or faiss-cpu)
  • scipy
  • scikit-learn
  • tqdm
  • cupy (for GPU acceleration)

Documentation

For detailed API documentation and examples, please visit our documentation site.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.