HumanCalib

Extrinsic calibration of a multi-camera rig from a person walking through it. No checkerboard, no wand: HumanCalib estimates the pose of every camera from the human pose seen by all of them, then gives the rig a metric scale and a vertical axis from the subject's height. The output is a Pose2Sim-format calibration, ready for markerless motion capture.

How it works

1. Pose estimation in every view with MeTRAbs, which predicts a metric 3D skeleton per camera.
2. Person selection: the walking subject is kept in every camera, bystanders are discarded.
3. Linear initialisation by aligning the per-camera 3D skeletons (Procrustes).
4. Bundle adjustment of all cameras on the 2D keypoints.
5. Metric scale and vertical from the subject's height and walk.

Details, design choices and what was measured to justify them: docs/METHOD.md.

Validation

Five public datasets, 77 trials, default settings. The three reported here each provide a laboratory-grade reference calibration and enough trials to summarise.

Camera geometry. Relative rotation between camera pairs, against the dataset's own calibration, and reprojection error of our own reconstruction, which needs no reference. Median over trials [min–max]:

Pixels are not comparable between rigs of different focal lengths; in angular terms the same errors are 2.4, 2.1 and 1.8 mrad. A low reprojection error means the calibration is not broken, not that it is accurate: on OpenCap, five smartphones on a tight arc and a walk of about two seconds cap the accuracy while leaving the residual low.

What it changes for the biomechanist. The same Pose2Sim chain was run twice per trial on the same 2D detections, with the laboratory calibration and with HumanCalib's, changing nothing else. The table compares the joint angles the two runs produce. Equivalence is declared when the upper bound of the 95 % confidence interval stays below the margin, for each of the 9 degrees of freedom (pelvis, hip, knee, ankle, subtalar):

Changing the calibration therefore moves the reported angles by about half a degree to one degree, below the 2° margin usually accepted in clinical gait analysis, and roughly ten times less than the 4–11° that separates such a chain from optical motion capture on the same trials.

• No failed calibration out of 77 (with RTMPose + VideoPose3D instead of MeTRAbs: 30).
• Metric scale within 1.1 % (median, on the four datasets not used to set it).

The two remaining datasets are reported in the paper: LBMC, whose two treadmill trials are too few to summarise, and COMFI, where HumanCalib proved closer to the laboratory's own motion capture than that dataset's own calibration, which makes any comparison against that reference a measure of the reference rather than of HumanCalib.

A paper is in preparation. The evaluation protocol is in docs/EVALUATION_PROTOCOL.md.

Installation

An NVIDIA GPU is needed for pose estimation (driver ≥ 525). Linux, WSL2 and Windows are supported.

pip — Linux or WSL2

Python 3.10 or 3.11. The CUDA libraries come from pip, nothing else to install.

python -m venv .venv && source .venv/bin/activate
pip install "humancalib[gpu] @ https://github.com/flodelaplace/HumanCalib/archive/refs/tags/v0.3.1.zip"

Windows

Python 3.10, in a conda environment that provides CUDA: TensorFlow 2.10 is the last version with GPU support on native Windows. CUDA 11.8 also covers recent GPUs (RTX 40xx), which CUDA 11.2 does not.

conda create -n humancalib -c conda-forge python=3.10 cudatoolkit=11.8 cudnn=8.9
conda activate humancalib
pip install "humancalib[gpu] @ https://github.com/flodelaplace/HumanCalib/archive/refs/tags/v0.3.1.zip"

Check the install

The demo videos are in the repository: unzip the same archive (or git clone the repository), then

humancalib run HumanCalib-0.3.1/demo HumanCalib-0.3.1/demo/Calib_scene.toml output/demo --height 1.78

It worked if the log shows Compute device: GPU and ends with an MRE summary table, and output/demo/results/Calib_scene_calibrated.toml exists. The first run downloads the MeTRAbs model (~700 MB, with a progress bar, resumed if interrupted) into ~/.cache/tfhub_modules; set TFHUB_CACHE_DIR to put it elsewhere.

From Python (e.g. inside Pose2Sim)

from humancalib import calibrate

toml = calibrate("session/videos", "session/Calib_intrinsics.toml", "session/humancalib",
                 height=1.78)

calibrate runs the same pipeline as humancalib run, with the same defaults, and returns the path of the calibrated TOML in Pose2Sim format. Any command-line option can be passed by name (extract_fps=25, ref_frame=120...); a failure raises humancalib.CalibrationError.

Docker

The exact environment the published results were obtained with. Requires Docker with Compose v2 and the NVIDIA Container Toolkit.

git clone https://github.com/flodelaplace/HumanCalib.git
cd HumanCalib
printf 'HOST_UID=%s\nHOST_GID=%s\n' "$(id -u)" "$(id -g)" > .env   # results owned by you, not root
docker compose build                                                 # ~2 GB download, resumable
docker compose run --rm calib demo

The model is kept in a Docker volume after the first run.

conda, exact environment

Every version pinned, as validated (TensorFlow 2.12, CUDA 11.8 from conda):

git clone https://github.com/flodelaplace/HumanCalib.git
cd HumanCalib
conda env create -f envs/calib.yaml
conda activate humancalib
pip install --no-deps -e .                 # adds the `humancalib` command, keeps the pins

The pip installs above use TensorFlow 2.15 (Linux) and 2.10 (Windows); they give the same calibration as this environment: on the demo, 0.02° median difference in relative rotation and 1 mm in camera positions; on a BioCV trial, the same error against the laboratory calibration (0.43°).

Without a GPU, pip install "humancalib @ https://github.com/flodelaplace/HumanCalib/archive/refs/tags/v0.3.1.zip" installs calibration, bundle adjustment, evaluation and scaling from existing pose files. The optional RTMPose + VideoPose3D backend has its own environment: see HOWTO.md.

Calibrating your own rig

1. Record. Synchronised videos from static cameras, while one person walks across the capture volume for a few passes.

2. Prepare a session folder with one video per camera and a Calib_scene.toml holding each camera's intrinsics in Pose2Sim format:

[camera01]
name = "camera01"
size = [1920.0, 1080.0]
matrix = [[1057.46, 0.0, 942.23], [0.0, 1056.83, 535.6], [0.0, 0.0, 1.0]]
distortions = [-0.041, 0.0086, -0.0002, 0.0002]
fisheye = false

Video names (without extension) must match the TOML sections. Good intrinsics matter more than anything else: see input/README.md.

3. Run.

humancalib run input/my_session input/my_session/Calib_scene.toml output/my_session \
    --height 1.84

# Docker: same arguments, with the container's paths
docker compose run --rm calib \
    /input/my_session /input/my_session/Calib_scene.toml /output/my_session --height 1.84

--height is the subject's height in metres; it sets the metric scale. The origin and horizontal axis come from a frame where every camera sees the head and both heels, chosen automatically (or --ref_frame N). On video faster than 50 Hz, add --extract_fps 25: same calibration, 2 to 8 times faster. Every option is described in HOWTO.md.

4. Results, in output/my_session/results/:

Documentation

Repository layout

src/humancalib/     the Python package: cli.py (the `humancalib` command), core/, pose/,
                    calibration/, pipeline/, postprocessing/, evaluation/
tests/              pytest suite, runs on a CPU in seconds
envs/               exact conda environments (calib, rtmpose, ci)
Dockerfile, compose.yaml, docker/    container images and entry point
demo/               4-camera demo session
docs/               documentation and research notes
input/, output/     your sessions and results (not tracked)

Licensing

The HumanCalib code is MIT. The pretrained pose models are not free for commercial use:

The Docker images contain no MeTRAbs weights unless built with BAKE_MODELS=1. This summary is not legal advice; check the upstream licences for your use.

Citation

Use GitHub's Cite this repository button (CITATION.cff). Please also cite the method HumanCalib builds on and the pose estimator:

• S.-E. Lee, K. Shibata, S. Nonaka, S. Nobuhara, K. Nishino. Extrinsic Camera Calibration From a Moving Person. IEEE Robotics and Automation Letters 7(4), 2022. doi:10.1109/LRA.2022.3192629 — original code
• I. Sárándi, T. Linder, K. O. Arras, B. Leibe. MeTRAbs: Metric-Scale Truncation-Robust Heatmaps for Absolute 3D Human Pose Estimation. IEEE T-BIOM, 2021.