Point Tracking for 4DGS

This repo is the point tracking Python extension for 4D Gaussian Splatting. It wraps sequence point trackers such as CoTracker3, VGGT, MV-TAP, and OpenD4RT behind one small registry, then applies them either to plain image sequences or to multi-timestep Gaussian Splatting camera datasets.

The package provides two common workflows:

• track sampled 2D points through one ordered image sequence
• project 3D Gaussians into a reference timestep, track those projected points across all timesteps, and attach the resulting tracks back to Gaussian Splatting camera datasets

Features

• Organised as a standard Python package with pip install support
• Shared point tracker registry with cotracker3, vggt, vggsfm, mvtap, d4rt, and blender implementations
• Single-view image sequence tracking and rendering
• Multi-timestep Gaussian Splatting camera dataset tracking
• Camera dataset reordering against a selected reference timestep
• Blender rigid-mesh tracking for scenes rendered by NVS2COLMAP
• Port the motion estimation workflow from TrackerSplat for point-tracker-driven motion synthesis
• Regularize 3DGS training with point tracker trajectories

Install

Prerequisites

• PyTorch (CUDA build recommended)
• CUDA Toolkit matching your PyTorch installation
• gaussian-splatting
• CoTracker
• VGGT for the optional vggt tracker

Install the core Gaussian Splatting dependency used by the 4DGS dataset utilities:

pip install wheel setuptools
pip install --upgrade gaussian-splatting

If you have trouble with gaussian-splatting, try installing it from source:

pip install wheel setuptools
pip install --upgrade git+https://github.com/yindaheng98/gaussian-splatting.git@master --no-build-isolation

Install tracker dependencies used by this package:

pip install --upgrade git+https://github.com/facebookresearch/co-tracker.git@main
pip install --upgrade Pillow PyYAML hydra-core omegaconf
pip install --upgrade git+https://github.com/facebookresearch/vggt.git@main
pip install --upgrade git+https://github.com/jytime/LightGlue.git#egg=lightglue

PyPI Install

pip install --upgrade track-4dgs

or build latest from source:

pip install wheel setuptools
pip install --upgrade git+https://github.com/yindaheng98/track-4dgs.git@master --no-build-isolation

Development Install

git clone --recursive https://github.com/yindaheng98/track-4dgs.git
cd track-4dgs
pip install --target . --upgrade . --no-deps

Download Checkpoints

CoTracker3 offline checkpoint:

mkdir -p checkpoints
wget -P checkpoints https://huggingface.co/facebook/cotracker3/resolve/main/scaled_offline.pth

VGGT commercial checkpoint:

mkdir -p checkpoints
wget -P checkpoints https://huggingface.co/facebook/VGGT-1B-Commercial/resolve/main/vggt_1B_commercial.pt --header="Authorization: Bearer $HF_TOKEN"

If the VGGT checkpoint is not present, VGGTPointTracker falls back to VGGT.from_pretrained("facebook/VGGT-1B").

VGGSfM tracker checkpoint:

mkdir -p checkpoints
wget -P checkpoints https://huggingface.co/facebook/VGGSfM/resolve/main/vggsfm_v2_tracker.pt

If the checkpoint is not present, load_vggsfm downloads it from that URL.

MATCHA checkpoint (from the MATCHA release):

mkdir -p checkpoints
# save the downloaded file as checkpoints/matcha_pretrained.pth

MV-TAP checkpoint (from the MV-TAP release):

mkdir -p checkpoints
# save the downloaded file as checkpoints/mvtap.ckpt

OpenD4RT checkpoint and model config:

# 48-frame model
mkdir -p checkpoints/OpenD4RT_48CLIP_9Mix_NoCropAUG
wget -P checkpoints/OpenD4RT_48CLIP_9Mix_NoCropAUG https://huggingface.co/Lijiaxin0111/OpenD4RT/resolve/main/checkpoints/OpenD4RT_48CLIP_9Mix_NoCropAUG/opend4rt.ckpt
wget -P checkpoints/OpenD4RT_48CLIP_9Mix_NoCropAUG https://huggingface.co/Lijiaxin0111/OpenD4RT/resolve/main/checkpoints/OpenD4RT_48CLIP_9Mix_NoCropAUG/model.yaml

# 32-frame model
mkdir -p checkpoints/OpenD4RT_32CLIP_9Dataset_NoAUG
wget -P checkpoints/OpenD4RT_32CLIP_9Dataset_NoAUG https://huggingface.co/Lijiaxin0111/OpenD4RT/resolve/main/checkpoints/OpenD4RT_32CLIP_9Dataset_NoAUG/opend4rt.ckpt
wget -P checkpoints/OpenD4RT_32CLIP_9Dataset_NoAUG https://huggingface.co/Lijiaxin0111/OpenD4RT/resolve/main/checkpoints/OpenD4RT_32CLIP_9Dataset_NoAUG/model.yaml

Each model requires both opend4rt.ckpt and its adjacent model.yaml. Pass either the checkpoint file or its containing directory through the checkpoint tracker option.

Command-Line Usage

List Registered Point Trackers

Verify that track_4dgs can import and register trackers:

python -c "import track_4dgs; print(track_4dgs.get_available_point_trackers())"

The built-in trackers are:

• cotracker3: CoTracker3 offline point tracker
• vggt: VGGT TrackHead point tracker
• mvtap: MV-TAP multi-view point tracker
• d4rt: OpenD4RT 2D correspondence head
• blender: rigid-mesh tracker for a .blend file rendered by NVS2COLMAP. See Blender mesh tracking.

Track One Image Sequence

python -m track_4dgs.track1v \
    -s data/frame_000.png data/frame_001.png data/frame_002.png \
    -d output/track1v-cotracker3 \
    --tracker cotracker3 \
    --num-points 256

The command samples random query points on the first image, tracks them through all source images, and saves a rendered track overlay sequence under the destination directory.

Tracker-specific options can be passed with repeated -m/--option_tracker values:

python -m track_4dgs.track1v \
    -s data/frame_000.png data/frame_001.png data/frame_002.png \
    -d output/track1v-vggt \
    --tracker vggt \
    -m checkpoint="'checkpoints/vggt_1B_commercial.pt'" \
    -m iters=4

Track 4DGS Camera Datasets

python -m track_4dgs.track2d \
    -s data/sequence/frame_000 data/sequence/frame_001 data/sequence/frame_002 \
    -d output/sequence \
    -i 30000 \
    --tracker cotracker3 \
    --init-dataset-index 0 \
    --num-points 256

track2d loads a trained Gaussian point cloud from:

<destination>/point_cloud/iteration_<iteration>/point_cloud.ply

It projects Gaussian centers into every camera of the reference timestep, tracks those projected points across the same camera views in all timesteps, and saves visualisations under:

<destination>/ours_<iteration>/track2d-<tracker>/

If cameras were saved outside the source scene folders, pass one camera path per source:

python -m track_4dgs.track2d \
    -s data/sequence/frame_000 data/sequence/frame_001 \
    -d output/sequence \
    -i 30000 \
    --load_cameras output/frame_000/cameras.json output/frame_001/cameras.json \
    --mode camera

Blender Mesh Tracking

track_4dgs.blender (--tracker blender) binds query pixels to rigid meshes in a .blend file and carries them with each mesh matrix_world. Use it together with the Blender tools in NVS2COLMAP.

NVS2COLMAP renders that blend file into a Neural 3D Video scene (poses_bounds.npy and one video per camera), then nvs2colmap.n3dv writes one COLMAP camera dataset per frame. This tracker reads the same .blend file and those frame datasets. Cameras stay in the blend world, look along +Z, and treat image +Y as downward, which is the OpenCV camera NVS2COLMAP stores in poses_bounds.npy. Dataset frame i in --sources is Blender frame frame_start + i * frame_step. Leaving frame_start unset uses scene.frame_start from the blend file.

Install NVS2COLMAP and keep a Blender executable on PATH as blender, or pass another path with -m blender="'path/to/blender'". warp-lang is already installed with this package.

nvs2colmap-sample-cameras can place the cameras and write poses_bounds.npy. Render the blend file from those cameras:

python -m nvs2colmap.blender \
    --path data/scene \
    --blend data/scene.blend \
    --n-frames 300

Decode the rendered videos into per-frame COLMAP datasets:

python -m nvs2colmap.n3dv \
    --path data/scene \
    --ffmpeg ffmpeg \
    --ffprobe ffprobe \
    --n-frames 300

Track those frames with the same blend file:

python -m track_4dgs.track2d \
    -s data/scene/frame1 data/scene/frame2 data/scene/frame3 \
    -d output/scene \
    -i 30000 \
    --tracker blender \
    -m blend="'data/scene.blend'"

The defaults line up when both NVS2COLMAP commands start at frame 1 and step by 1, and --sources lists frame1, frame2, ... in order. nvs2colmap.blender --frame-step already drops frames inside each camera video, so nvs2colmap.n3dv still writes consecutive frame* folders. If that render used --start-number N and --frame-step S, set frame_start to the blend file's scene.frame_start + (N - 1) and frame_step to S. A later nvs2colmap.n3dv --start-number M skips M - 1 rendered frames, so add (M - 1) * S to frame_start. For a blend file whose scene.frame_start is 1, --start-number 10 --frame-step 2 on the render and a default n3dv extract look like this:

python -m track_4dgs.track2d \
    -s data/scene/frame1 data/scene/frame2 data/scene/frame3 \
    -d output/scene \
    -i 30000 \
    --tracker blender \
    -m blend="'data/scene.blend'" \
    -m frame_start=10 \
    -m frame_step=2

API Usage

Build a Tracker

from track_4dgs.registry import build_point_tracker, get_available_point_trackers

print(get_available_point_trackers())

tracker = build_point_tracker(
    "cotracker3",
    checkpoint="checkpoints/scaled_offline.pth",
).to("cuda")

Track Image Tensors

import torch

from track_4dgs.track1v import load_image, sample_query

frames = [
    load_image("data/frame_000.png", "cuda"),
    load_image("data/frame_001.png", "cuda"),
]
query = sample_query(frames[0], num_points=256)

with torch.no_grad():
    track = tracker.track_view(query, frames, [None] * len(frames))

print(track.points.shape)      # [num_frames, num_points, 2]
print(track.visibility.shape)  # [num_frames, num_points]
print(track.confidence.shape)  # [num_frames, num_points]

Track Gaussian Splatting Camera Datasets

from gaussian_splatting.prepare import prepare_gaussians

from track_4dgs.prepare import prepare_datasets, prepare_tracker
from track_4dgs.track2d import query_views_from_gaussians

sources = [
    "data/sequence/frame_000",
    "data/sequence/frame_001",
]
datasets = prepare_datasets(
    sources=sources,
    device="cuda",
    reorder_reference_idx=0,
)
gaussians = prepare_gaussians(
    sh_degree=3,
    source=sources[0],
    device="cuda",
    load_ply="output/sequence/point_cloud/iteration_30000/point_cloud.ply",
)
dataset_tracker = prepare_tracker(
    tracker_name="cotracker3",
    device="cuda",
    tracker_configs={"checkpoint": "checkpoints/scaled_offline.pth"},
)

queries = query_views_from_gaussians(
    datasets=datasets,
    gaussians=gaussians,
    init_dataset_index=0,
    num_points=256,
)
tracked_datasets = dataset_tracker(queries, datasets)

camera_track = tracked_datasets[0][0].custom_data["track"]
print(camera_track.points.shape)

Design: Point Tracker Registry

AbstractPointTracker.__call__ validates inputs, then calls track. It takes a view-major Query (points [V, N, 2], frame_indices [V, N]) and frame-major camera datasets, and returns one Track per view. CameraDatasetTracker attaches those tracks back onto the camera datasets:

query + frames -> validate + track -> view_tracks -> tracked camera datasets

Single-view trackers (AbstractViewPointTracker, e.g. CoTracker3 / VGGT) split the query by view and implement track_batch(points, frame_indices, frames, masks) for one camera sequence. Joint trackers (AbstractBatchPointTracker, e.g. MV-TAP) consume all views at once and batch along the point dimension.

Frame 0 cameras --\
Frame 1 cameras ----> single-view loop, or one joint multi-view call ----> view_tracks
Frame 2 cameras --/

This keeps model-specific code isolated in tracker implementations while the 4DGS workflow can use any registered tracker by name.

Extending: Adding a New Point Tracker

Single-view trackers return Track(points=[D, N, 2], visibility=[D, N], confidence=[D, N], mask=[D, N]) from track_batch:

from collections.abc import Sequence

import torch

from track_4dgs.tracker import AbstractViewPointTracker, Track


class MyPointTracker(AbstractViewPointTracker):
    def to(self, device):
        self.device = torch.device(device)
        return self

    def track_batch(
        self,
        points: torch.Tensor,
        frame_indices: torch.Tensor,
        frames: Sequence[torch.Tensor],
        frame_masks: Sequence[torch.Tensor | None],
    ) -> Track:
        ...

Register it at import time:

from track_4dgs.registry import register_point_tracker

register_point_tracker("mytracker", MyPointTracker)

After registration, it is available everywhere:

python -m track_4dgs.track1v --tracker mytracker -s frame0.png frame1.png -d output/mytracker

Acknowledgement

This repo is developed based on CoTracker, VGGT, MV-TAP, LightGlue, gaussian-splatting (packaged), and NVS2COLMAP. Many thanks to the authors for open-sourcing their codebases.