diff-plonk 0.4

Diffusion Geolocalization package for PLONK models

Around the World in 80 Timesteps:
A Generative Approach to Global Visual Geolocation

Nicolas Dufour, David Picard, Vicky Kalogeiton, Loic Landrieu

Introducing the first generative geolocation method based on diffusion and flow matching! We learn the relationship between visual content and location by denoising random locations conditionally to images.

➜ New SOTA for visual geolocation on OpenStreetView-5M, YFCC-100M, and iNat-21

➜ Generate global probability density maps and quantify localizability

➜ Introduce the problem of probabilistic visual geolocation

This repository contains the code for the paper "Around the World in 80 Timesteps: A Generative Approach to Global Visual Geolocation"

Project website: https://nicolas-dufour.github.io/plonk

Arxiv: https://arxiv.org/abs/2412.06781

HuggingFace Collection: https://huggingface.co/collections/nicolas-dufour/around-the-world-in-80-timesteps-6758595d634129e6fc63dad9

Demo: https://huggingface.co/spaces/nicolas-dufour/PLONK

Installation

If you want to use our models, you can directly install the diff-plonk package:

conda create -n plonk python=3.10
conda activate plonk
pip install diff-plonk

For a local installation, if you want to train the model or use the demo, you can clone the repository and install the dependencies:

From the github repository:

git clone https://github.com/nicolas-dufour/plonk.git
cd plonk
conda create -n plonk python=3.10
conda activate plonk
pip install -e .

From pip:

conda create -n plonk python=3.10
conda activate plonk
pip install diff-plonk

Models

We provide pre-trained models for the OSV-5M, YFCC-100M, and iNat-21 datasets. You can download them from the huggingface hub.

Demo

We provide a demo of the model hosted on the following HuggingFace Space.

Running the demo locally

pip install -e ".[demo]"

And then run the following command:

streamlit run plonk/demo/demo.py 

Usage

To use the models, you can use our pipeline:

from plonk import PlonkPipeline

pipeline = PlonkPipeline("nicolas-dufour/PLONK_YFCC")

gps_coords = pipeline(images, batch_size=1024)

With images being a list of PIL images or a PIL image.

3 different models are provided for each dataset:

• nicolas-dufour/PLONK_OSV_5M: OSV-5M
• nicolas-dufour/PLONK_YFCC: YFCC-100M
• nicolas-dufour/PLONK_iNat: iNat-21

Baselines models

We also provide the baseline models for the OSV-5M, YFCC-100M, and iNat-21 datasets for the carthesian flow matching and diffusion models. You can download them from the huggingface hub.

Flow matching models:

• nicolas-dufour/PLONK_OSV_5M_flow: OSV-5M
• nicolas-dufour/PLONK_YFCC_flow: YFCC-100M
• nicolas-dufour/PLONK_iNat_flow: iNat-21

Diffusion models:

• nicolas-dufour/PLONK_OSV_5M_diffusion: OSV-5M
• nicolas-dufour/PLONK_YFCC_diffusion: YFCC-100M
• nicolas-dufour/PLONK_iNat_diffusion: iNat-21

Training

Install training dependencies

You will need to install the training dependencies:

pip install -e ".[train]"

Downloading the dataset

You will need to download the OSV5M webdataset from the https://huggingface.co/datasets/osv5m/osv5m-wds in the plonk/datasets/osv5m folder.

Training the model

To train the model, you can use the following command:

python plonk/train.py exp=osv_5m_geoadalnmlp_r3_small_sigmoid_flow_riemann mode=traineval experiment_name=My_OSV_5M_Experiment

Evaluating the model

To evaluate the model, you can use the following command:

python plonk/train.py exp=osv_5m_geoadalnmlp_r3_small_sigmoid_flow_riemann mode=eval experiment_name=My_OSV_5M_Experiment

YFCC and iNAturalist

For YFCC and iNat, script to preprocess the dataset are provided in the plonk/data/extract_embeddings and plonk/data/to_webdataset folders.

Citation

If you find this work useful for your research or use our code, please cite our paper:

@article{dufour2024around,
  title={Around the World in 80 Timesteps: A Generative Approach to Global Visual Geolocation},
  author={Dufour, Nicolas and Picard, David and Kalogeiton, Vicky and Landrieu, Loic},
  journal={arXiv preprint arXiv:2412.06781},
  year={2024}
}