LookBench: A Live and Holistic Open Benchmark for Fashion Image Retrieval

LookBench is a live, holistic, and challenging benchmark for fashion image retrieval in real e-commerce settings. This repository provides the official evaluation code and model implementations.

📰 News

• [2026-09] New ShopRank-Bench reranking track and the ZooWork-ShopRanker reranker family (0.6B / 4B / 8B) — paper released on arXiv
• [2026-01] LookBench paper released on arXiv
• [2026-01] GR-Lite open-source model released
• [2026-01] Initial benchmark dataset released

📖 Overview

LookBench addresses the limitations of existing fashion retrieval benchmarks by providing:

• 🔄 Continuously Refreshing Samples: Mitigates data contamination with time-stamped, periodically updated test sets
• 🎯 Diverse Retrieval Tasks: Covers single-item and multi-item retrieval across real studio, AI-generated studio, real street-look, and AI-generated street-look scenarios
• 📊 Attribute-Supervised Evaluation: Fine-grained evaluation based on 100+ fashion attributes across categories
• 🏆 Challenging Benchmarks: Many strong baselines achieve below 60% Recall@1

Benchmark Subsets

Reranking Track: ShopRank-Bench

LookBench also hosts a text reranking track. Where the retrieval track ranks a gallery against a query image, the reranking track asks a cross-encoder which of two candidate products a shopper would prefer.

ShopRank-Bench is a contamination-limited e-commerce preference benchmark: 10,511 pairs over 2,991 queries drawn from live shopping traffic, labelled by a cross-family LLM judge panel and tiered by how many families committed a verdict (1,843 gold / 4,445 silver / 4,223 bronze). Every pair ships in two formats — the canonical structured attribute schema and a natural-language rendering — carrying the same label, so a per-format accuracy gap measures sensitivity to serialization rather than a difference in labels.

• 🤗 Dataset: srpone/zoowork-shoprank-bench (release pending)
• 📄 Paper: ZooWork-ShopRanker: An Open, Preference-Aligned E-Commerce Reranker
• 🤗 Models: ZooWork-ShopRanker (0.6B / 4B / 8B)
• 🏆 Leaderboard: ShopRank-Bench track page
• 🌐 ZooWork: zoowork.ai

from look_bench.data import ShopRankPairs
from look_bench.models import get_model
from look_bench.metrics import PairwiseAccuracyEvaluator

# or ShopRankPairs.from_jsonl("preference.jsonl", ...) for a local copy of the release file
pairs = ShopRankPairs.from_hub(text_format="structured")
# Each ZooWork-ShopRanker repo holds the base weights and the LoRA adapter
_, reranker = get_model("qwen3-reranker").load_model(
    "srpone/zoowork-shopranker-4b", model_path="srpone/zoowork-shopranker-4b")

results = PairwiseAccuracyEvaluator().evaluate_reranker(
    reranker, list(pairs), preferred_key="preferred", rejected_key="rejected")
print(results["pairwise_accuracy"], results["pairwise_accuracy_gold"])

Reranking needs one extra dependency: pip install look-bench[reranking].

🚀 Quick Start

Installation

Option 1: Install from PyPI (Recommended)

pip install "look-bench>=0.4.0"            # retrieval
pip install "look-bench[reranking]>=0.4.0" # + ShopRank-Bench rerankers (peft)

Use 0.4.0 or later. Releases up to 0.3.0 installed top-level datasets, models, metrics and utils packages that overwrote Hugging Face datasets; everything now lives under the look_bench package (from look_bench.data import ...).

Option 2: Install from Source

# Clone the repository
git clone https://github.com/SerendipityOneInc/look-bench.git
cd look-bench

# Install in development mode
pip install -e .

# Or install dependencies only
pip install -r requirements.txt

Optional: Install with Examples Support

For running example notebooks and scripts that require matplotlib:

pip install look-bench[examples]

Load Dataset from Hugging Face

The LookBench dataset is hosted on Hugging Face and can be loaded directly:

Option 1: Using look-bench utility (Recommended)

from look_bench.utils import load_lookbench_dataset

# Load a specific config
dataset = load_lookbench_dataset("real_studio_flat")

# Access query and gallery splits
query_data = dataset['query']
gallery_data = dataset['gallery']

print(f"Query samples: {len(query_data)}")
print(f"Gallery samples: {len(gallery_data)}")

Option 2: Using Hugging Face datasets directly

from datasets import load_dataset

# Load a specific config
dataset = load_dataset("srpone/look-bench", "real_studio_flat")

# Access query and gallery splits
query_data = dataset['query']
gallery_data = dataset['gallery']

print(f"Query samples: {len(query_data)}")
print(f"Gallery samples: {len(gallery_data)}")

Quick Evaluation

import torch
from look_bench.manager import ConfigManager, ModelManager

# Load model
config_manager = ConfigManager()  # packaged look_bench/configs/config.yaml
model_manager = ModelManager(config_manager)

model, _ = model_manager.load_model('clip')
transform = model_manager.get_transform('clip')

# Extract features from an image
sample = dataset['real_studio_flat']['query'][0]
image_tensor = transform(sample['image']).unsqueeze(0)

if torch.cuda.is_available():
    model = model.cuda()
    image_tensor = image_tensor.cuda()

with torch.no_grad():
    features = model(image_tensor)

print(f"Feature shape: {features.shape}")

Run Full Evaluation

# Run evaluation with the packaged default configuration
look-bench

# Run a specific pipeline
look-bench --pipeline evaluation

# Use your own configuration (start from look_bench/configs/config.yaml)
look-bench --config my_config.yaml

# From a source checkout, `python main.py ...` is equivalent

Example Scripts & Notebooks

We provide both Python scripts and Google Colab notebooks for easy experimentation:

📓 Colab Notebooks (Run in Browser)

• 01_quickstart.ipynb - Basic usage and dataset exploration
• 02_model_evaluation.ipynb - Complete evaluation pipeline
• 03_custom_model.ipynb - Integrate custom models

🐍 Python Scripts (Run Locally)

• examples/00_data_exploration.py - Dataset exploration and statistics
• examples/01_load_grlite_model.py - Load and test GR-Lite model
• examples/02_model_evaluation.py - Complete model evaluation pipeline
• examples/03_custom_model.py - Integrate your own custom models

# Run examples locally
python examples/00_data_exploration.py
python examples/01_load_grlite_model.py
python examples/02_model_evaluation.py
python examples/03_custom_model.py

🏗️ Architecture

look-bench/
├── main.py                     # Source-checkout entry point (= `look-bench` command)
└── look_bench/                 # The installed package
    ├── main.py                 # CLI (config-driven)
    ├── manager.py              # Configuration, model, and data managers; DEFAULT_CONFIG
    ├── configs/config.yaml     # Packaged default configuration
    ├── runner/                 # Pipeline execution framework
    │   ├── base_pipeline.py    # Base pipeline class
    │   ├── evaluator.py        # Core evaluation logic
    │   ├── pipeline.py         # Pipeline registry
    │   ├── evaluation_pipeline.py          # Standard evaluation pipeline
    │   └── feature_extraction_pipeline.py  # Feature extraction pipeline
    ├── models/                 # Model implementations and registry
    │   ├── base.py             # Base (embedding) model interface
    │   ├── reranker_base.py    # Base reranker interface
    │   ├── registry.py         # Model registration system
    │   ├── factory.py          # Model factory
    │   ├── clip_model.py / siglip_model.py / dinov2_model.py / grlite_model.py
    │   └── qwen3_reranker.py   # Qwen3-Reranker (optional LoRA)
    ├── data/                   # Dataset loading (BEIR-style) and ShopRank-Bench pairs
    │   ├── base.py             # Base dataset implementation
    │   ├── registry.py         # Dataset registry
    │   └── shoprank_dataset.py # ShopRankPairs
    ├── metrics/                # Evaluation metrics
    │   ├── rank.py / mrr.py / ndcg.py / map.py
    │   ├── pairwise.py         # Pairwise accuracy (reranking)
    │   └── significance.py     # Paired tests
    └── utils/                  # Utilities and logging

🎯 Supported Models

Reranking models score a (query, candidate) pair directly rather than producing an embedding, so they register through BaseReranker instead of BaseModel:

⚙️ Configuration

Edit look_bench/configs/config.yaml to configure models and evaluation settings:

# Pipeline configuration
pipeline:
  name: "evaluation"  # evaluation, feature_extraction
  model: "clip"
  dataset: "fashion200k"
  args: {}

# Model configuration
clip:
  enabled: true
  model_name: "openai/clip-vit-base-patch16"
  input_size: 224
  embedding_dim: 512
  device: "cuda"

# Evaluation settings
evaluation:
  metric: "recall"
  top_k: [1, 5, 10, 20]
  l2norm: true

📊 Evaluation Metrics

LookBench supports multiple evaluation metrics:

• Recall@K: Top-K retrieval accuracy (K=1, 5, 10, 20)
• MRR: Mean Reciprocal Rank
• NDCG@K: Normalized Discounted Cumulative Gain
• MAP: Mean Average Precision

Fine-Grained Evaluation

All metrics are computed with attribute-level matching:

• Fine Recall@1: Requires exact category and all attributes to match
• Coarse Recall@1: Only requires category to match
• nDCG@K: Graded relevance based on attribute overlap

🔧 Advanced Usage

Custom Model Integration

LookBench makes it easy to integrate your own models using the registry pattern. Here's a quick example:

from look_bench.models.base import BaseModel
from look_bench.models.registry import register_model
import torch.nn as nn
from torchvision import models, transforms

@register_model("resnet50", metadata={
    "description": "ResNet-50 for fashion retrieval",
    "framework": "PyTorch",
    "input_size": 224,
    "embedding_dim": 2048
})
class ResNet50Model(BaseModel):
    @classmethod
    def load_model(cls, model_name: str, model_path: str = None):
        model = models.resnet50(pretrained=True)
        model = nn.Sequential(*list(model.children())[:-1])  # Remove FC layer
        
        # Wrapper to flatten output
        class Wrapper(nn.Module):
            def __init__(self, backbone):
                super().__init__()
                self.backbone = backbone
            def forward(self, x):
                return self.backbone(x).squeeze(-1).squeeze(-1)
        
        return Wrapper(model), cls()
    
    @classmethod
    def get_transform(cls, input_size: int = 224):
        return transforms.Compose([
            transforms.Resize((input_size, input_size)),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.485, 0.456, 0.406],
                               std=[0.229, 0.224, 0.225])
        ])

Then add your model to look_bench/configs/config.yaml:

resnet50:
  enabled: true
  model_name: "resnet50"
  model_path: null  # or path to your weights
  input_size: 224
  embedding_dim: 2048
  device: "cuda"

For complete examples, see examples/03_custom_model.py

Custom Pipeline

Create custom evaluation pipelines:

from look_bench.runner.base_pipeline import BasePipeline
from look_bench.runner.pipeline import register_pipeline

@register_pipeline("custom_pipeline")
class CustomPipeline(BasePipeline):
    def get_pipeline_name(self) -> str:
        return "custom_pipeline"
    
    def run(self, **kwargs):
        # Your custom logic here
        model_name = kwargs.get('model_name', 'clip')
        dataset_type = kwargs.get('dataset_type', 'fashion200k')
        
        # Load model and data
        model, _ = self.model_manager.load_model(model_name)
        # ... your evaluation logic
        
        return {"status": "success", "results": results}

📈 Results

Fine Recall@1 Performance

Our GR-Lite model achieves state-of-the-art performance on LookBench. Fine Recall@1 requires exact category and all attributes to match:

Coarse Recall@1 Performance

Coarse Recall@1 only requires category match (more lenient):

nDCG@5 Performance

nDCG@5 evaluates ranking quality with graded relevance based on attribute overlap:

See our paper for complete results including MRR and additional models.

📄 Citation

If you use LookBench in your research, please cite:

@article{gao2026lookbench,
  title={LookBench: A Live and Holistic Open Benchmark for Fashion Image Retrieval}, 
  author={Chao Gao and Siqiao Xue and Yimin Peng and Jiwen Fu and Tingyi Gu and Shanshan Li and Fan Zhou},
  year={2026},
  url={https://arxiv.org/abs/2601.14706}, 
  journal={arXiv preprint arXiv:2601.14706},
}

If you use the ShopRank-Bench reranking track or the ZooWork-ShopRanker models, please also cite:

@article{xue2026zooworkshopranker,
  title   = {ZooWork-ShopRanker: An Open, Preference-Aligned E-Commerce Reranker},
  author  = {Xue, Siqiao and Liu, Shuxuan and Hu, Ning},
  journal = {arXiv preprint arXiv:2609.31002},
  year    = {2026},
  url     = {https://arxiv.org/abs/2609.31002}
}

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

The GR-Lite model weights are distributed under the DINOv3 License as they are derived from Meta's DINOv3 model.