Perform a markerless kinematic analysis from multiple calibrated views as a unified workflow from an OpenPose input to an OpenSim result.
Project description
Pose2Sim
Pose2Sim
provides a workflow for 3D markerless kinematics, as an alternative to the more usual marker-based motion capture methods.
Pose2Sim stands for "OpenPose to OpenSim", as it uses OpenPose inputs (2D keypoints coordinates obtained from multiple videos) and leads to an OpenSim result (full-body 3D joint angles).
Contents
Installation and Demonstration
Installation
-
Install OpenPose (instructions there).
Windows portable demo is enough. -
Install OpenSim 4.x (there).
Tested up to v4.4-beta on Windows. Has to be compiled from source on Linux (see there). -
Optional. Install Anaconda or Miniconda.
Open an Anaconda terminal and create a virtual environment with typing:conda create -n Pose2Sim python=3.8.8 conda activate Pose2Sim
-
Install Pose2Sim:
If you don't use Anaconda, typepython -V
in terminal to make sure python>=3.6 is installed.-
OPTION 1: Quick install: Open a terminal.
pip install pose2sim
-
OPTION 2: Build from source: Open a terminal in the directory of your choice and Clone the Pose2Sim repository.
git clone https://gitlab.inria.fr/perfanalytics/pose2sim.git cd pose2sim pip install .
-
Demonstration Part-1: Build 3D TRC file on Python
This demonstration provides an example experiment of a person balancing on a beam, filmed with 4 calibrated cameras processed with OpenPose.
Open a terminal and check package location with pip show pose2sim | grep Location
.
Copy this path and go to the Demo folder with cd <path>\pose2sim\Demo
.
Type python
, and test the following code:
from Pose2Sim import Pose2Sim
Pose2Sim.calibrateCams()
Pose2Sim.track2D()
Pose2Sim.triangulate3D()
Pose2Sim.filter3D()
You should obtain a plot of all the 3D coordinates trajectories. You can check the logs inDemo\Users\logs.txt
.
Results are stored as .trc files in the Demo/pose-3d
directory.
N.B.: Default parameters have been provided in Demo\Users\Config.toml
but can be edited.
Demonstration Part-2: Obtain 3D joint angles with OpenSim
In the same vein as you would do with marker-based kinematics, start with scaling your model, and then perform inverse kinematics.
Scaling
- Open OpenSim.
- Open the provided
Model_Pose2Sim_Body25b.osim
model frompose2sim/Demo/opensim
. (File -> Open Model) - Load the provided
Scaling_Setup_Pose2Sim_Body25b.xml
scaling file frompose2sim/Demo/opensim
. (Tools -> Scale model -> Load) - Run. You should see your skeletal model take the static pose.
Inverse kinematics
- Load the provided
IK_Setup_Pose2Sim_Body25b.xml
scaling file frompose2sim/Demo/opensim
. (Tools -> Inverse kinematics -> Load) - Run. You should see your skeletal model move in the Vizualizer window.
Use on your own data
Deeper explanations and instructions are given below.
Prepare for running on your own data
Get ready.
-
Find your
Pose2Sim\Empty_project
, copy-paste it where you like and give it the name of your choice. -
Edit the
User\Config.toml
file as needed, especially regarding the path to your project. -
Populate the
raw-2d
folder with your camera images or videos.Project │ ├──opensim │ ├──Geometry │ ├──Model_Pose2Sim_Body25b.osim │ ├──Scaling_Setup_Pose2Sim_Body25b.xml │ └──IK_Setup_Pose2Sim_Body25b.xml │ ├── raw-2d │ ├──raw_cam1_img │ ├──... │ └──raw_camN_img │ └──User └──Config.toml
2D pose estimation
Estimate 2D pose from images with Openpose.
Open a command prompt in your OpenPose directory.
Launch OpenPose for each raw image folder:
bin\OpenPoseDemo.exe --model_pose BODY_25B --image_dir <PATH_TO_PROJECT_DIR>\raw-2d\raw_cam1_img --write_json <PATH_TO_PROJECT_DIR>\pose-2d\pose_cam1_json
- N.B.: The BODY_25B model has more accurate results; however, feel free to use any OpenPose model (BODY_25B, BODY_25, COCO, with face and/or hands, etc), and to work with videos instead of image files.
- N.B.: You can also use DeepLabCut, or other 2D pose estimators instead.
If you decide to do so, you'll have to (1) translate the format to json files (withDLC_to_OpenPose.py
script, see Utilities); (2) report the model keypoints in the 'skeleton.py' file; (3) create an OpenSim model if you need 3D joint angles.
N.B.: Markers are not needed and are used only for validation
The project hierarchy becomes: (CLICK TO SHOW)
Project │ ├──opensim │ ├──Geometry │ ├──Model_Pose2Sim_Body25b.osim │ ├──Scaling_Setup_Pose2Sim_Body25b.xml │ └──IK_Setup_Pose2Sim_Body25b.xml │ ├──pose-2d │ ├──pose_cam1_json │ ├──... │ └──pose_camN_json │ ├── raw-2d │ ├──raw_cam1_img │ ├──... │ └──raw_camN_img │ └──User └──Config.toml
Cameras calibration
Calibrate your cameras.
- If you already have a calibration file (.qca.txt from Qualisys for example):
- copy it in the
calib-2d
folder - set [calibration] type to 'qca' in your
Config.toml
file.
or
- If you have taken pictures or videos of a checkerboard with your cameras:
- create a folder for each camera in your
calib-2d
folder, - copy there the images or videos of the checkerboard
- set [calibration] type to 'checkerboard' in your
Config.toml
file, and adjust other parameters.
Open an Anaconda prompt or a terminal.
By default, calibrateCams()
will look for Config.toml
in the User
folder of your current directory. If you want to store it somewhere else (e.g. in your data directory), specify this path as an argument: Pose2Sim.calibrateCams(r'path_to_config.toml')
.
from Pose2Sim import Pose2Sim
Pose2Sim.calibrateCams()
Output:
The project hierarchy becomes: (CLICK TO SHOW)
Project │ ├──calib-2d │ ├──calib_cam1_img │ ├──... │ ├──calib_camN_img │ └──Calib.toml │ ├──opensim │ ├──Geometry │ ├──Model_Pose2Sim_Body25b.osim │ ├──Scaling_Setup_Pose2Sim_Body25b.xml │ └──IK_Setup_Pose2Sim_Body25b.xml │ ├──pose-2d │ ├──pose_cam1_json │ ├──... │ └──pose_camN_json │ ├── raw-2d │ ├──raw_cam1_img │ ├──... │ └──raw_camN_img │ └──User └──Config.toml
2D tracking of person
Track the person viewed by the most cameras, in case of several detections by OpenPose.
N.B.: Skip this step if only one person is in the field of view.
Open an Anaconda terminal
By default, track2D()
will look for Config.toml
in the User
folder of your current directory. If you want to store it somewhere else (e.g. in your data directory), specify this path as an argument: Pose2Sim.track2D(r'path_to_config.toml')
.
from Pose2Sim import Pose2Sim
Pose2Sim.track2D()
Check printed output. If results are not satisfying, try and release the constraints in the Config.toml
file.
Output:
The project hierarchy becomes: (CLICK TO SHOW)
Project │ ├──calib-2d │ ├──calib_cam1_img │ ├──... │ ├──calib_camN_img │ └──Calib.toml │ ├──opensim │ ├──Geometry │ ├──Model_Pose2Sim_Body25b.osim │ ├──Scaling_Setup_Pose2Sim_Body25b.xml │ └──IK_Setup_Pose2Sim_Body25b.xml │ ├──pose-2d │ ├──pose_cam1_json │ ├──... │ └──pose_camN_json │ ├──pose-2d-tracked │ ├──tracked_cam1_json │ ├──... │ └──tracked_camN_json │ ├── raw-2d │ ├──raw_cam1_img │ ├──... │ └──raw_camN_img │ └──User └──Config.toml
3D triangulation
Triangulate your 2D coordinates in a robust way.
Open an Anaconda terminal.
By default, triangulate3D()
will look for Config.toml
in the User
folder of your current directory. If you want to store it somewhere else (e.g. in your data directory), specify this path as an argument: Pose2Sim.triangulate3D(r'path_to_config.toml')
.
from Pose2Sim import Pose2Sim
Pose2Sim.triangulate3D()
Check printed output, and vizualise your trc in OpenSim.
If your triangulation is not satisfying, try and release the constraints in the Config.toml
file.
Output:
The project hierarchy becomes: (CLICK TO SHOW)
Project │ ├──calib-2d │ ├──calib_cam1_img │ ├──... │ ├──calib_camN_img │ └──Calib.toml │ ├──opensim │ ├──Geometry │ ├──Model_Pose2Sim_Body25b.osim │ ├──Scaling_Setup_Pose2Sim_Body25b.xml │ └──IK_Setup_Pose2Sim_Body25b.xml │ ├──pose-2d │ ├──pose_cam1_json │ ├──... │ └──pose_camN_json │ ├──pose-2d-tracked │ ├──tracked_cam1_json │ ├──... │ └──tracked_camN_json │ ├──pose-3d └──Pose-3d.trc> │ ├── raw-2d │ ├──raw_cam1_img │ ├──... │ └──raw_camN_img │ └──User └──Config.toml
3D Filtering
Filter your 3D coordinates.
Open an Anaconda terminal.
By default, filter3D()
will look for Config.toml
in the User
folder of your current directory. If you want to store it somewhere else (e.g. in your data directory), specify this path as an argument: Pose2Sim.filter3D(r'path_to_config.toml')
.
from Pose2Sim import Pose2Sim
Pose2Sim.filter3D()
Check your filtration with the displayed figures, and vizualise your trc in OpenSim. If your filtering is not satisfying, try and change the parameters in the Config.toml
file.
Output:
The project hierarchy becomes: (CLICK TO SHOW)
Project │ ├──calib-2d │ ├──calib_cam1_img │ ├──... │ ├──calib_camN_img │ └──Calib.toml │ ├──opensim │ ├──Geometry │ ├──Model_Pose2Sim_Body25b.osim │ ├──Scaling_Setup_Pose2Sim_Body25b.xml │ └──IK_Setup_Pose2Sim_Body25b.xml │ ├──pose-2d │ ├──pose_cam1_json │ ├──... │ └──pose_camN_json │ ├──pose-2d-tracked │ ├──tracked_cam1_json │ ├──... │ └──tracked_camN_json │ ├──pose-3d │ ├──Pose-3d.trc │ └──Pose-3d-filtered.trc │ ├── raw-2d │ ├──raw_cam1_img │ ├──... │ └──raw_camN_img │ └──User └──Config.toml
OpenSim kinematics
Obtain 3D joint angles.
Scaling
- Use the previous steps to capture a static pose, typically an A-pose or a T-pose.
- Open OpenSim.
- Open the provided
Model_Pose2Sim_Body25b.osim
model frompose2sim/Empty_project/opensim
. (File -> Open Model) - Load the provided
Scaling_Setup_Pose2Sim_Body25b.xml
scaling file frompose2sim/Empty_project/opensim
. (Tools -> Scale model -> Load) - Replace the example static .trc file with your own data.
- Run
- Save the new scaled OpenSim model.
Inverse kinematics
- Use Pose2Sim to generate 3D trajectories.
- Open OpenSim.
- Load the provided
IK_Setup_Pose2Sim_Body25b.xml
scaling file frompose2sim/Empty_project/opensim
. (Tools -> Inverse kinematics -> Load) - Replace the example .trc file with your own data, and specify the path to your angle kinematics output file.
- Run
- Motion results will appear as .mot file in the
pose2sim/Empty_project/opensim
directory (automatically saved).
Command line
Alternatively, you can use command-line tools:
Open an Anaconda terminal in your OpenSim/bin directory, typically C:\OpenSim <Version>\bin
.
You'll need to adjust the time_range
, output_motion_file
, and the paths to the .osim and .trc files in your setup file.
opensim-cmd run-tool <PATH_TO_POSE2SIM>/OpenSim/Setup/<YOUR SCALING OR IK SETUP FILE>.xml
The project hierarchy becomes: (CLICK TO SHOW)
Project │ ├──calib-2d │ ├──calib_cam1_img │ ├──... │ ├──calib_camN_img │ └──Calib.toml │ ├──opensim │ ├──Geometry │ ├──Model_Pose2Sim_Body25b.osim │ ├──Scaling_Setup_Pose2Sim_Body25b.xml │ ├──Model_Pose2Sim_Body25b_Scaled.osim │ ├──IK_Setup_Pose2Sim_Body25b.xml │ └──IK_result.mot │ ├──pose-2d │ ├──pose_cam1_json │ ├──... │ └──pose_camN_json │ ├──pose-2d-tracked │ ├──tracked_cam1_json │ ├──... │ └──tracked_camN_json │ ├──pose-3d │ ├──Pose-3d.trc │ └──Pose-3d-filtered.trc │ ├── raw-2d │ ├──raw_cam1_img │ ├──... │ └──raw_camN_img │ └──User └──Config.toml
Utilities
A list of standalone tools, which can be both run as scripts or imported as functions. Check usage in the docstrings of each python file.\
Converting files and Calibrating (CLICK TO SHOW)
DLC_to_OpenPose.py
Converts a DeepLabCut (h5) 2D pose estimation file into OpenPose (json) files.
c3d_to_trc.py
Converts 3D point data of a .c3d file to a .trc file compatible with OpenSim. No analog data (force plates, emg) nor computed data (angles, powers, etc) are retrieved.
calib_from_checkerboard.py
Calibrates cameras with images or a video of a checkerboard, saves calibration in a Pose2Sim .toml calibration file.
calib_qca_to_toml.py
Converts a Qualisys .qca.txt calibration file to the Pose2Sim .toml calibration file (similar to what is used in AniPose).
calib_toml_to_qca.py
Converts a Pose2Sim .toml calibration file (e.g., from a checkerboard) to a Qualisys .qca.txt calibration file.
calib_yml_to_toml.py
Converts OpenCV intrinsic and extrinsic .yml calibration files to an OpenCV .toml calibration file.
calib_toml_to_yml.py
Converts an OpenCV .toml calibration file to OpenCV intrinsic and extrinsic .yml calibration files.
Plotting tools (CLICK TO SHOW)
json_display_with_img.py
Overlays 2D detected json coordinates on original raw images. High confidence keypoints are green, low confidence ones are red.
json_display_without_img.py
Plots an animation of 2D detected json coordinates.
trc_plot.py
Displays X, Y, Z coordinates of each 3D keypoint of a TRC file in a different matplotlib tab.
Other trc tools (CLICK TO SHOW)
trc_desample.py
Undersamples a trc file.
trc_Zup_to_Yup.py
Changes Z-up system coordinates to Y-up system coordinates.
trc_filter.py
Filters trc files. Available filters: Butterworth, Butterworth on speed, Gaussian, LOESS, Median.
trc_gaitevents.py
Detects gait events from point coordinates according to Zeni et al. (2008).
trc_combine.py
Combine two trc files, for example a triangulated DeepLabCut trc file and a triangulated OpenPose trc file.
How to cite and how to contribute
How to cite
If you use this code or data, please cite Pagnon et al., 2022 or Pagnon et al., 2021.
@Article{Pagnon_2022_Pose2Sim,
AUTHOR = {Pagnon, David and Domalain, Mathieu and Reveret, Lionel},
TITLE = {Pose2Sim: An End-to-End Workflow for 3D Markerless Sports Kinematics—Part 2: Accuracy},
JOURNAL = {Sensors},
YEAR = {2022},
PUBLISHER = {Multidisciplinary Digital Publishing Institute},
URL = {https://www.mdpi.com/1424-8220/22/7/2712}
}
@Article{Pagnon_2021_Pose2Sim,
AUTHOR = {Pagnon, David and Domalain, Mathieu and Reveret, Lionel},
TITLE = {Pose2Sim: An End-to-End Workflow for 3D Markerless Sports Kinematics—Part 1: Robustness},
JOURNAL = {Sensors},
YEAR = {2021},
PUBLISHER = {Multidisciplinary Digital Publishing Institute},
URL = {https://www.mdpi.com/1424-8220/21/19/6530},
}
How to contribute
I would happily welcome any proposal for new features, code improvement, and more!
If you want to contribute to Pose2Sim, please follow this guide on how to fork, modify and push code, and submit a pull request. I would appreciate it if you provided as much useful information as possible about how you modified the code, and a rationale for why you're making this pull request. Please also specify on which operating system and on which python version you have tested the code.
Here is a to-do list, for general guidance purposes only:
Integrate as a Blender and / or Maya add-on. See Maya-Mocap and BlendOSim Multiple persons kinematics (triangulating multiple persons, and sorting them in time) Use aniposelib for better calibration Copy-paste muscles from OpenSim lifting full-body model for inverse dynamics and more Finish deploying Body_135, MediaPipe, AlphaPose, and DeepLabCut compatibility
Conda package and Docker image Solve limb swapping Implement normalized DLT and RANSAC triangulation, as well as a triangulation refinement step (cf DOI:10.1109/TMM.2022.3171102) Implement optimal fixed-interval Kalman smoothing Utilities: convert Vicon xcp calibration file to toml Run from command line via click or typer Catch errors Make GUI
BSD 3-Clause License
Copyright (c) 2022, perfanalytics All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
-
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
-
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
-
Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.