This repository extends the standard COCO person keypoint evaluation by implementing the Extended-OKS (Ex-OKS) metric introduced in the ProbPose paper. Built on top of the original xtcocotools and the official COCO API, Ex-OKS remains fully backward-compatible with the standard OKS. It adds support for:

• Out-of-image keypoints (points annotated outside the image boundary or activation window) to asses model's robustness
• Per-visibility-level mAP breakdowns to pinpoint which keypoints cause errors

Why ExOCocoTools? ExO stands both for Ex-OKS and for greek "exo", meaning "outside". This evaluation protocol is the first to evaluate on keypoints outside of the image.

Table of Contents

• Extended-OKS vs. OKS
• Visibility Levels
• Usage / Demo
• Installation
• Acknowledgements and Citation

Extended-OKS vs. OKS

• OKS (Object Keypoint Similarity) measures similarity between predicted and ground-truth keypoints within the image.
• Ex-OKS (Extended OKS) extends OKS by:
  • Penalizing in-image predictions when the ground-truth is out-of-image
  • Penalizing out-of-image predictions when the groud-truth is in-image
  • The same as OKS when both ground-truth and prediction are in-image

Detailed Explanation

Formally, Ex-OKS has the same form as OKS -- euclidean distance scaled by image scale and per-keypoint sigma.

    \text{Ex-OKS} = \exp{(\frac{-d_{i}^2}{2k^2\sigma^2})}

However, for Ex-OKS, the distance $d_i$ depends on the situation (ground-truth in/out; precition in/out). Below, you can see formal defintion along with illustrative scheme.

d_i = \begin{cases}
        d_e(x^{*}_i, x'_i) & \text{if } p^{*}_p = 1 \text{ and } p'_p = 1 \\
        d_e(\text{AW}, x'_i) & \text{if } p^{*}_p = 0 \text{ and } p'_p = 1 \\
        d_e(x^{*}_i, \text{AW}) & \text{if } p^{*}_p = 1 \text{ and } p'_p = 0 \\
        0                               & \text{else} 
    \end{cases} \\

In summary, Ex-OKS extends OKS to situation when ground-truth or prediction are outside of the activation window. For more details, read the full explanation in the ProbPose paper.

Visibility Levels

Apart from Ex-OKS, this library also shows mAP for different visibility levels. Best demonstrated in CropCOCO Demo, rows with specific visibility show performance for such keypoints.

The output above shows that for

• occluded keypoints (v=1), the mAP is 38.9
• for visible keypoints (v=2), the mAP is 79.1
• and for out-of-image keypoints (v=3), the mAP is 31.7

Usage / Demo

from exococotools.coco import COCO
from exococotools.cocoeval import COCOeval

# Standard OKS evaluation (backward-compatible)
cocoEval = COCOeval(cocoGt_json, cocoDt_json, iouType='keypoints', extended_oks=False)

# --- OR ---

# Extended-OKS evaluation
cocoEval = COCOeval(cocoGt_json, cocoDt_json, iouType='keypoints', extended_oks=True)

# Evaluate and print results
cocoEvalExt.evaluate()
cocoEvalExt.accumulate()
cocoEvalExt.summarize()

For more details, see COCO Demo or CropCOCO Demo files.

Installation

From PyPI

pip install exococotools

From Source

git clone https://github.com/MiraPurkrabek/Ex-cocotools
cd Ex-cocotools
pip install -r requirements.txt
pip install -e .

Acknowledgements and Citation

This implementation builds upon the COCO API and xtcocotools projects. The Extended-OKS metric and its evaluation methodology are described in the ProbPose paper:

@InProceedings{Purkrabek2025CVPR,
    author    = {Purkrabek, Miroslav and Matas, Jiri},
    title     = {ProbPose: A Probabilistic Approach to 2D Human Pose Estimation},
    booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
    month     = {June},
    year      = {2025},
    pages     = {27124-27133}
}