🔍 VisualMatch

AI-powered visual product search engine using OpenAI CLIP

Upload a product photo → get the matching item with a purchase link, instantly.

✨ What It Does

VisualMatch uses OpenAI CLIP (openai/clip-vit-base-patch32) to match user-uploaded product photos against an e-commerce catalog. It integrates with Instagram Direct via NovinHub webhooks, enabling customers to find and buy products directly from DMs.

Key features:

• 🧠 Real CLIP embeddings — 512-dimensional vectors, cosine similarity search
• 📸 Instagram DM bot — Conversational visual search via NovinHub webhooks
• ⚡ <100ms search latency — Brute-force vector search over pre-computed embeddings
• 🌐 Web dashboard — Built-in admin UI for catalog management and search testing
• 🐳 Docker ready — One-command deployment with Docker Compose
• 🔌 Extensible — Fan-out webhook router supports multiple bot services

🏗️ Architecture

visualmatch/
├── app/                          # Application package
│   ├── main.py                   # FastAPI server & webhook handler
│   ├── config.py                 # Environment-driven configuration
│   ├── models/                   # Pydantic data models
│   │   └── webhook.py
│   └── services/                 # Core business logic
│       ├── clip_engine.py        # CLIP embedding extraction
│       ├── catalog_service.py    # Vector search over catalog
│       └── novinhub_client.py    # NovinHub API client
├── scripts/                      # Data pipeline utilities
│   ├── generate_live_catalog.py  # Fetch products from live API
│   └── import_products.py        # Import from Excel spreadsheet
├── tests/                        # Pytest test suite
├── web/                          # Web dashboard (HTML/CSS/JS)
├── docs/                         # Documentation
│   ├── architecture.md           # System design & diagrams
│   └── api.md                    # API reference
├── pyproject.toml                # Python project configuration
├── Dockerfile                    # Multi-stage container build
├── docker-compose.yml            # Container orchestration
├── Makefile                      # Development commands
└── .github/workflows/ci.yml     # CI pipeline

See docs/architecture.md for detailed system diagrams and component documentation.

🚀 Quick Start

Prerequisites

• Python 3.10+
• (Optional) Docker

Local Setup

# Clone the repository
git clone https://github.com/your-org/visualmatch.git
cd visualmatch

# Create virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
make install
# Or manually: pip install -e ".[ml,import]"

# Configure environment
cp .env.example .env
# Edit .env — at minimum, set HF_TOKEN for CLIP model access

# Generate the product catalog (fetches from your API + computes embeddings)
make catalog-generate

# Start the server
make dev

The server starts at http://localhost:8000:

• 📖 API docs: http://localhost:8000/docs
• 🖥️ Web dashboard: http://localhost:8000/web/

Docker

cp .env.example .env
# Edit .env with your settings

docker compose up --build

📡 Webhook Setup

To connect with Instagram Direct via NovinHub:

1. Create a NovinHub account and connect your Instagram Business page
2. Set your webhook URL to: https://your-domain.com/api/v1/novinhub-webhook
3. Configure your .env file:

   NOVINHUB_TOKEN=your_novinhub_api_token
   NOVINHUB_CHANNEL_ID=your_channel_id
   NOVINHUB_ACCOUNT_OWNER_ID=your_instagram_social_user_id
   

4. Set up an Icebreaker button in NovinHub with payload FIND_MODEL

🔧 Catalog Generation

The system needs a product catalog with pre-computed CLIP embeddings. Two methods are available:

From Live API (Recommended)

Fetches products from your e-commerce API, downloads product images, and computes CLIP embeddings:

make catalog-generate
# Or: python -m scripts.generate_live_catalog

Note: You'll need to update the API URLs in scripts/generate_live_catalog.py to point to your own product API.

From Excel Spreadsheet

Place your product Excel file in data/ and run:

make catalog-import
# Or: python -m scripts.import_products

Excel format — Required columns:

⚙️ Configuration

All configuration is managed through environment variables. See .env.example for the complete list.

🧪 Testing

# Run full test suite
make test

# With coverage
make test-cov

# Lint & type check
make lint

📖 API Reference

Full API documentation is available at:

• Interactive: http://localhost:8000/docs (Swagger UI)
• Static: docs/api.md

Key Endpoints

🤝 Contributing

Contributions are welcome! Please read CONTRIBUTING.md for development setup, code style, and the pull request process.

📄 License

This project is licensed under the MIT License.