[CVPR24] Polos: Multimodal Metric Learning from Human Feedback for Image Captioning
Project description
🌟 Polos: Multimodal Metric Learning from Human Feedback for Image Captioning
- Accepted at CVPR 2024
- project page
- arXiv
Establishing an automatic evaluation metric that closely aligns with human judgements is essential for the effective development of image captioning models. Data-driven metrics have recently gained prominence in this field, demonstrating a stronger correlation with human judgements than classic metrics such as CIDEr and SPICE. However, these approaches pose challenges; for instance, they lack sufficient capabilities to handle hallucinations and to generalize across various types of images and texts. This limitation is partly attributed to the fact that existing approaches compute scalar similarities merely using embeddings learned from tasks that are not directly related to image captioning evaluation. In this study, we propose Polos, a supervised automatic evaluation metric tailored for image captioning models. To enhance robustness and practicality, we also present Multimodal Metric Learning from Human Feedback (M LHF), a novel framework for developing metrics based on human feedback. In line with the principles of M LHF, Polos is trained directly from human feedback and computes evaluation scores using multimodal inputs, employing a parallel feature extraction mechanism that leverages SimCSE and CLIP. This mechanism enables our metric to effectively model intricate relationships within the vector space of text-image pairs as well as text-text pairs. In addition, we have constructed a large-scale dataset for M LHF, which comprises 131K human judgements collected from 550 evaluators. Our dataset further distinguishes itself from existing datasets in terms of the inclusion of diverse captions, which are collected from humans and generated from ten image captioning models, including modern models. Our approach has achieved state-of-the-art performance on various image captioning benchmarks, including Composite, Flickr8K-Expert, Flickr8K-CF, FOIL, and our dataset, demonstrating its effectiveness and robustness.
Instructions
We assume the following environment for our experiments:
- Python 3.10.0 (pyenv is strongly recommended)
- Poetry for dependency management (refer to Poetry documentation)
- PyTorch version 2.1.0 with CUDA 11.8 support
- PyTorch Lightning for model training facilitation
Clone & Install
git clone git@github.com:keio-smilab24/Polos.git
cd Polos
pyenv virtualenv 3.10.0 polos
pyenv local polos
sh install.sh # cuda=11.8
Datasets
- Polaris
- The Polaris dataset can be downloaded at this link.
- Unzip and extract the contents into the
data_en
directory.
- Flickr8k
- Composite / PASCAL-50S / FOIL
- For the Composite, PASCAL-50S, and FOIL datasets, download them from the following links:
- Composite
- PASCAL-50S
- FOIL
Checkpoint
The best checkpoint can be downloaded at this link. Unzip and extract the checkpoints
.
Train
sh train.sh
Evaluation
PAC-S checkpoints are required to assess PAC-S.
Download the checkpoints according to the instructions on the authors' github and place them in the specified locations.
sh validate.sh
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.