GVHMR: World-Grounded Human Motion Recovery via Gravity-View Coordinates

Project Page | Paper

World-Grounded Human Motion Recovery via Gravity-View Coordinates
Zehong Shen^*, Huaijin Pi^*, Yan Xia, Zhi Cen, Sida Peng^†, Zechen Hu, Hujun Bao, Ruizhen Hu, Xiaowei Zhou
SIGGRAPH Asia 2024

News 🔥

• [2025-03-08] By default not using DPVO. We implemented a SimpleVO, which is more efficient and compatible with GVHMR.
• [2025-03-08] We added a new option f_mm to specify the focal length of the fullframe camera in mm.

Setup

Please see installation for details.

Quick Start

Google Colab demo for GVHMR

HuggingFace demo for GVHMR

Demo

Demo entries are provided in tools/demo. Use -s to skip visual odometry if you know the camera is static, otherwise the camera will be estimated by DPVO. We also provide a script demo_folder.py to inference a entire folder.

python tools/demo/demo.py --video=docs/example_video/tennis.mp4 -s
python tools/demo/demo_folder.py -f inputs/demo/folder_in -d outputs/demo/folder_out -s

The same CLI is available after pip install as python -m hmr4d.demo or the gvhmr-demo console script.

Use as a library

The demo pipeline is importable, so GVHMR can be run programmatically (e.g. as an Eden extension). The model is loaded once and reused across calls:

from hmr4d.demo import GVHMR

# `checkpoint_root` points at your downloaded checkpoints (body models, GVHMR/HMR2/
# ViTPose/YOLO/DPVO weights). It defaults to ./inputs/checkpoints; set it (or the
# $GVHMR_CHECKPOINT_ROOT env var) to run from any working directory.
model = GVHMR(checkpoint_root="/path/to/inputs/checkpoints")
out = model.recover("docs/example_video/tennis.mp4", static_cam=True)  # render=True for overlay videos
# out -> {"smpl_params_global", "smpl_params_incam", "K_fullimg", "output_dir"}

GVHMR is CUDA-only (select a GPU with CUDA_VISIBLE_DEVICES). It composes its own Hydra config, so it must run in a process without an already-initialized Hydra — when embedding it in another Hydra-using application (such as Eden), run it out-of-process.

Reproduce

1. Test: To reproduce the 3DPW, RICH, and EMDB results in a single run, use the following command:

   python tools/train.py global/task=gvhmr/test_3dpw_emdb_rich exp=gvhmr/mixed/mixed ckpt_path=inputs/checkpoints/gvhmr/gvhmr_siga24_release.ckpt
   

   To test individual datasets, change global/task to gvhmr/test_3dpw, gvhmr/test_rich, or gvhmr/test_emdb.

2. Train: To train the model, use the following command:

   # The gvhmr_siga24_release.ckpt is trained with 2x4090 for 420 epochs, note that different GPU settings may lead to different results.
   python tools/train.py exp=gvhmr/mixed/mixed
   

   During training, note that we do not employ post-processing as in the test script, so the global metrics results will differ (but should still be good for comparison with baseline methods).

Citation

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{shen2024gvhmr,
  title={World-Grounded Human Motion Recovery via Gravity-View Coordinates},
  author={Shen, Zehong and Pi, Huaijin and Xia, Yan and Cen, Zhi and Peng, Sida and Hu, Zechen and Bao, Hujun and Hu, Ruizhen and Zhou, Xiaowei},
  booktitle={SIGGRAPH Asia Conference Proceedings},
  year={2024}
}

Acknowledgement

We thank the authors of WHAM, 4D-Humans, and ViTPose-Pytorch for their great works, without which our project/code would not be possible.