Left: Emerging matching capabilities from self-supervision. Simply using Poincar3's attention map reliable tracks can be created. Produced by demo.py. Right: Henri Poincaré argued that a being without motion cannot understand 3D space.

Overview

We release a model, Poincar3, that learns multi-view geometry from only training on image sequences. Poincar3 uses a multi-view transformer and a self-distillation objective giving strong zero-shot features and attention maps.

Install

uv add poincar3          # or: pip install poincar3

The model itself needs only torch. Extras pull in what the scripts need:

uv add "poincar3[demo]"     # demo.py
uv add "poincar3[train]"    # training, mvcorr and SeeSE3 evals
uv add "poincar3[eval]"     # + matchbench and feed-forward reconstruction

To work from a clone instead (tested on Linux with Python 3.12):

uv sync --extra eval

Usage

The checkpoint auto-downloads on first use:

import torch
from poincar3 import Poincar3

model = Poincar3().eval().cuda()

# A multi-view batch: [batch, frames, 3, H, W], RGB in [0, 1], H and W multiples of 16.
images = torch.rand(1, 4, 3, 448, 448).cuda()

with torch.no_grad():
    patch_logits, patch_features, global_logits, camera_tokens = model(images)

# patch_features:  [1, 4, 784, 1024]  dense per-frame tokens, cross-view attended
# camera_tokens:   [1, 4, 1024]       one per-frame scene/pose token

Demo

We illustrate how to create attention tracks by simply running:

uv run python demo.py

We also provide code for plotting the raw feature correlations in cross_correlations.py.

Pretrained weights

Auto-downloaded on first use. To fetch it directly:

• Poincar3 ViT-L/16 — Download

Evaluation

All evaluations, except feed-forward reconstruction, can be accessed through the experiments/eval.py endpoint. You can get ScanNet and NAVI following these instructions. To get the possible configurations, simply run it with the flag --help. For example, you run multi-view correspondence estimation on scannet by:

uv run python experiments/eval.py --evaluation mvcorr --mvcorr.dataset scannet

Feed-forward reconstruction

Camera-pose and per-pixel-depth heads on top of the backbone, under two protocols:

# full finetune
torchrun --nproc-per-node 4 experiments/ffrecon/train.py --name finetune-poincar3

# frozen backbone + a small adapter
torchrun --nproc-per-node 4 experiments/ffrecon/train_adapter.py --backbone poincar3

# relative pose evaluation
uv run python experiments/ffrecon/eval.py --checkpoint <path> --evaluation relpose --relpose.dataset megadepth
# point-cloud estimation
uv run python experiments/ffrecon/eval.py --checkpoint <path> --evaluation pointcloud --pointcloud.dataset eth3d

Training

torchrun --nproc-per-node 4 experiments/train.py --name my-run

Training data

While we trained on large collection of 3D datasets, we illustrate our training protocol on ScanNet++ and RealEstate10K. You can download them using their official download links.

License

MIT, except where a file notes otherwise. src/poincar3/layers/ and parts of heads/ derive from DINOv3 and VGGT and carry their original licenses; benchmarks/mv_consistency/ is adapted from probe3d (MIT).

Acknowledgement

Built on DINOv3, DINOv2, VGGT, probe3d and RoMa.

BibTeX

TBD