RTMPose without mmcv, mmpose, mmdet etc.
Project description
rtmlib
rtmlib is a super lightweight library to conduct pose estimation based on RTMPose models WITHOUT any dependencies like mmcv, mmpose, mmdet, etc.
Basically, rtmlib only requires these dependencies:
- numpy
- opencv-python
- opencv-contrib-python
Optionally, you can use other common backends like pytorch, onnxruntime, tensorrt to accelerate the inference process.
Installation
git clone https://github.com/Tau-J/rtmlib.git
cd rtmlib
pip install -r requirements.txt
pip install -e .
# [optional]
# pip install onnxruntime
# or
# pip install onnxruntime-gpu
TODO
- Support MMPose-style skeleton visualization
- Support OpenPose-style skeleton visualization
- Support WholeBody
- Support ONNXRuntime backend
- Support auto download and cache models
- Lightweight models
- Support alias to choose model
- Support PoseTracker proposed in RTMPose
- Support TensorRT backend
- Gradio interface
- Compatible with Controlnet
Model Zoo
By defaults, rtmlib will automatically download and apply models with the best performace. But you can also specify the model you want to use by passing the onnx_model
argument.
More models can be found in RTMPose Model Zoo.
Detectors
Person
Notes:
- Models trained on HumanArt can detect both real human and cartoon characters.
- Models trained on COCO can only detect real human.
ONNX Model | Input Size | Description |
---|---|---|
YOLOX-l | 640x640 | trained on COCO val2017 |
YOLOX-nano | 416x416 | trained on HumanArt |
YOLOX-tiny | 416x416 | trained on HumanArt |
YOLOX-s | 640x640 | trained on HumanArt |
YOLOX-m | 640x640 | trained on HumanArt |
YOLOX-l | 640x640 | trained on HumanArt |
YOLOX-x | 640x640 | trained on HumanArt |
Pose Estimators
Body
ONNX Model | Input Size | Description |
---|---|---|
RTMPose-t | 256x192 | Body 17 Keypoints |
RTMPose-s | 256x192 | Body 17 Keypoints |
RTMPose-m | 256x192 | Body 17 Keypoints |
RTMPose-l | 384x288 | Body 17 Keypoints |
RTMPose-x | 384x288 | Body 17 Keypoints |
WholeBody
ONNX Model | Input Size | Description |
---|---|---|
RTMW-x | 384x288 | Wholebody 133 Keypoints |
Demo
Run webui.py
:
# Please make sure you have installed gradio
# pip install gradio
python webui.py
Here is also a simple demo to show how to use rtmlib to conduct pose estimation on a single image.
import cv2
from rtmlib import Wholebody, draw_skeleton
device = 'cpu'
backend = 'onnxruntime' # opencv, onnxruntime
img = cv2.imread('./demo.jpg')
openpose_skeleton = False # True for openpose-style, False for mmpose-style
wholebody = Wholebody(to_openpose=openpose_skeleton,
backend=backend, device=device)
keypoints, scores = wholebody(img)
# visualize
# if you want to use black background instead of original image,
# img_show = np.zeros(img_show.shape, dtype=np.uint8)
img_show = draw_skeleton(img_show, keypoints, scores, kpt_thr=0.5)
cv2.imshow('img', img_show)
cv2.waitKey()
Visualization
MMPose-style | OpenPose-style |
---|---|
Citation
@misc{rtmlib,
title={rtmlib},
author={Tao Jiang},
year={2023},
howpublished = {\url{https://github.com/Tau-J/rtmlib}},
}
@misc{https://doi.org/10.48550/arxiv.2303.07399,
doi = {10.48550/ARXIV.2303.07399},
url = {https://arxiv.org/abs/2303.07399},
author = {Jiang, Tao and Lu, Peng and Zhang, Li and Ma, Ningsheng and Han, Rui and Lyu, Chengqi and Li, Yining and Chen, Kai},
keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose},
publisher = {arXiv},
year = {2023},
copyright = {Creative Commons Attribution 4.0 International}
}
Acknowledgement
Our code is based on these repos:
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.