Point Tracking for 4DGS

This repo is the point tracking Python extension for 4D Gaussian Splatting. It wraps sequence point trackers such as CoTracker3, VGGT, MV-TAP, and OpenD4RT behind one small registry, then applies them either to plain image sequences or to multi-timestep Gaussian Splatting camera datasets.

The package provides two common workflows:

• track sampled 2D points through one ordered image sequence
• project 3D Gaussians into a reference timestep, track those projected points across all timesteps, and attach the resulting tracks back to Gaussian Splatting camera datasets

Features

• Organised as a standard Python package with pip install support
• Shared point tracker registry with cotracker3, vggt, mvtap, and d4rt implementations
• Single-view image sequence tracking and rendering
• Multi-timestep Gaussian Splatting camera dataset tracking
• Camera dataset reordering against a selected reference timestep
• Port the motion estimation workflow from TrackerSplat for point-tracker-driven motion synthesis
• Regularize 3DGS training with point tracker trajectories

Install

Prerequisites

• PyTorch (CUDA build recommended)
• CUDA Toolkit matching your PyTorch installation
• gaussian-splatting
• CoTracker
• VGGT for the optional vggt tracker

Install the core Gaussian Splatting dependency used by the 4DGS dataset utilities:

pip install wheel setuptools
pip install --upgrade gaussian-splatting

If you have trouble with gaussian-splatting, try installing it from source:

pip install wheel setuptools
pip install --upgrade git+https://github.com/yindaheng98/gaussian-splatting.git@master --no-build-isolation

Install tracker dependencies used by this package:

pip install --upgrade git+https://github.com/facebookresearch/co-tracker.git@main
pip install --upgrade Pillow PyYAML hydra-core omegaconf
pip install --upgrade git+https://github.com/facebookresearch/vggt.git@main
pip install --upgrade git+https://github.com/jytime/LightGlue.git#egg=lightglue

PyPI Install

pip install --upgrade track-4dgs

or build latest from source:

pip install wheel setuptools
pip install --upgrade git+https://github.com/yindaheng98/track-4dgs.git@master --no-build-isolation

Development Install

git clone --recursive https://github.com/yindaheng98/track-4dgs.git
cd track-4dgs
pip install --target . --upgrade . --no-deps

Download Checkpoints

CoTracker3 offline checkpoint:

mkdir -p checkpoints
wget -P checkpoints https://huggingface.co/facebook/cotracker3/resolve/main/scaled_offline.pth

VGGT commercial checkpoint:

mkdir -p checkpoints
wget -P checkpoints https://huggingface.co/facebook/VGGT-1B-Commercial/resolve/main/vggt_1B_commercial.pt --header="Authorization: Bearer $HF_TOKEN"

If the VGGT checkpoint is not present, VGGTPointTracker falls back to VGGT.from_pretrained("facebook/VGGT-1B").

MV-TAP checkpoint (from the MV-TAP release):

mkdir -p checkpoints
# save the downloaded file as checkpoints/mvtap.ckpt

OpenD4RT checkpoint and model config:

# 48-frame model
mkdir -p checkpoints/OpenD4RT_48CLIP_9Mix_NoCropAUG
wget -P checkpoints/OpenD4RT_48CLIP_9Mix_NoCropAUG https://huggingface.co/Lijiaxin0111/OpenD4RT/resolve/main/checkpoints/OpenD4RT_48CLIP_9Mix_NoCropAUG/opend4rt.ckpt
wget -P checkpoints/OpenD4RT_48CLIP_9Mix_NoCropAUG https://huggingface.co/Lijiaxin0111/OpenD4RT/resolve/main/checkpoints/OpenD4RT_48CLIP_9Mix_NoCropAUG/model.yaml

# 32-frame model
mkdir -p checkpoints/OpenD4RT_32CLIP_9Dataset_NoAUG
wget -P checkpoints/OpenD4RT_32CLIP_9Dataset_NoAUG https://huggingface.co/Lijiaxin0111/OpenD4RT/resolve/main/checkpoints/OpenD4RT_32CLIP_9Dataset_NoAUG/opend4rt.ckpt
wget -P checkpoints/OpenD4RT_32CLIP_9Dataset_NoAUG https://huggingface.co/Lijiaxin0111/OpenD4RT/resolve/main/checkpoints/OpenD4RT_32CLIP_9Dataset_NoAUG/model.yaml

Each model requires both opend4rt.ckpt and its adjacent model.yaml. Pass either the checkpoint file or its containing directory through the checkpoint tracker option.

Command-Line Usage

List Registered Point Trackers

Verify that track_4dgs can import and register trackers:

python -c "import track_4dgs; print(track_4dgs.get_available_point_trackers())"

The built-in trackers are:

• cotracker3: CoTracker3 offline point tracker
• vggt: VGGT TrackHead point tracker
• mvtap: MV-TAP multi-view point tracker
• d4rt: OpenD4RT 2D correspondence head

Track One Image Sequence

python -m track_4dgs.track1v \
    -s data/frame_000.png data/frame_001.png data/frame_002.png \
    -d output/track1v-cotracker3 \
    --tracker cotracker3 \
    --num-points 256

The command samples random query points on the first image, tracks them through all source images, and saves a rendered track overlay sequence under the destination directory.

Tracker-specific options can be passed with repeated -m/--option_tracker values:

python -m track_4dgs.track1v \
    -s data/frame_000.png data/frame_001.png data/frame_002.png \
    -d output/track1v-vggt \
    --tracker vggt \
    -m checkpoint="'checkpoints/vggt_1B_commercial.pt'" \
    -m iters=4

Track 4DGS Camera Datasets

python -m track_4dgs.track2d \
    -s data/sequence/frame_000 data/sequence/frame_001 data/sequence/frame_002 \
    -d output/sequence \
    -i 30000 \
    --tracker cotracker3 \
    --init-dataset-index 0 \
    --num-points 256

track2d loads a trained Gaussian point cloud from:

<destination>/point_cloud/iteration_<iteration>/point_cloud.ply

It projects Gaussian centers into every camera of the reference timestep, tracks those projected points across the same camera views in all timesteps, and saves visualisations under:

<destination>/ours_<iteration>/track2d-<tracker>/

If cameras were saved outside the source scene folders, pass one camera path per source:

python -m track_4dgs.track2d \
    -s data/sequence/frame_000 data/sequence/frame_001 \
    -d output/sequence \
    -i 30000 \
    --load_cameras output/frame_000/cameras.json output/frame_001/cameras.json \
    --mode camera

API Usage

Build a Tracker

from track_4dgs.registry import build_point_tracker, get_available_point_trackers

print(get_available_point_trackers())

tracker = build_point_tracker(
    "cotracker3",
    checkpoint="checkpoints/scaled_offline.pth",
).to("cuda")

Track Image Tensors

import torch

from track_4dgs.track1v import load_image, sample_query

frames = [
    load_image("data/frame_000.png", "cuda"),
    load_image("data/frame_001.png", "cuda"),
]
query = sample_query(frames[0], num_points=256)

with torch.no_grad():
    track = tracker.track_view(query, frames, [None] * len(frames))

print(track.points.shape)      # [num_frames, num_points, 2]
print(track.visibility.shape)  # [num_frames, num_points]
print(track.confidence.shape)  # [num_frames, num_points]

Track Gaussian Splatting Camera Datasets

from gaussian_splatting.prepare import prepare_gaussians

from track_4dgs.prepare import prepare_datasets, prepare_tracker
from track_4dgs.track2d import query_views_from_gaussians

sources = [
    "data/sequence/frame_000",
    "data/sequence/frame_001",
]
datasets = prepare_datasets(
    sources=sources,
    device="cuda",
    reorder_reference_idx=0,
)
gaussians = prepare_gaussians(
    sh_degree=3,
    source=sources[0],
    device="cuda",
    load_ply="output/sequence/point_cloud/iteration_30000/point_cloud.ply",
)
dataset_tracker = prepare_tracker(
    tracker_name="cotracker3",
    device="cuda",
    tracker_configs={"checkpoint": "checkpoints/scaled_offline.pth"},
)

queries = query_views_from_gaussians(
    datasets=datasets,
    gaussians=gaussians,
    init_dataset_index=0,
    num_points=256,
)
tracked_datasets = dataset_tracker(queries, datasets)

camera_track = tracked_datasets[0][0].custom_data["track"]
print(camera_track.points.shape)

Design: Point Tracker Registry

AbstractPointTracker.__call__ validates inputs, then calls track. It takes a view-major Query (points [V, N, 2], frame_indices [V, N]) and frame-major camera datasets, and returns one Track per view. CameraDatasetTracker attaches those tracks back onto the camera datasets:

query + frames -> validate + track -> view_tracks -> tracked camera datasets

Single-view trackers (AbstractViewPointTracker, e.g. CoTracker3 / VGGT) split the query by view and implement track_batch(points, frame_indices, frames, masks) for one camera sequence. Joint trackers (AbstractBatchPointTracker, e.g. MV-TAP) consume all views at once and batch along the point dimension.

Frame 0 cameras --\
Frame 1 cameras ----> single-view loop, or one joint multi-view call ----> view_tracks
Frame 2 cameras --/

This keeps model-specific code isolated in tracker implementations while the 4DGS workflow can use any registered tracker by name.

Extending: Adding a New Point Tracker

Single-view trackers return Track(points=[D, N, 2], visibility=[D, N], confidence=[D, N], mask=[D, N]) from track_batch:

from collections.abc import Sequence

import torch

from track_4dgs.tracker import AbstractViewPointTracker, Track


class MyPointTracker(AbstractViewPointTracker):
    def to(self, device):
        self.device = torch.device(device)
        return self

    def track_batch(
        self,
        points: torch.Tensor,
        frame_indices: torch.Tensor,
        frames: Sequence[torch.Tensor],
        frame_masks: Sequence[torch.Tensor | None],
    ) -> Track:
        ...

Register it at import time:

from track_4dgs.registry import register_point_tracker

register_point_tracker("mytracker", MyPointTracker)

After registration, it is available everywhere:

python -m track_4dgs.track1v --tracker mytracker -s frame0.png frame1.png -d output/mytracker

Acknowledgement

This repo is developed based on CoTracker, VGGT, MV-TAP, LightGlue, and gaussian-splatting (packaged). Many thanks to the authors for open-sourcing their codebases.