poseaudit

How far off are the joint angles, tilts and lengths you read from a pose model? OKS and mAP say whether the points land near the right place; poseaudit says whether the angle you compute from them is good enough to use. It judges your model against your labels as a scale is validated: bias (the mean offset), limits of agreement (where 95% of the differences fall), ICC (agreement, 1 at best) and how often it is off by too much.

Joint angles already computed (Pose2Sim, Sports2D, OpenSim, your own code) against motion capture or a goniometer? poseaudit paired --table angles.csv --big-error 5 reads a long CSV (subject, trial, frame, measure, pred, ref, unit; --column ref=Goniometer for other names), and --pred-mot/--ref-mot read OpenSim .mot files, with intervals by subject (paired values).

What plain numpy misses

The mean |error| is one line of numpy. The work is around it (brackets are 95% intervals):

• Who was skipped. Keep only the detections yolo11n-pose scored 0.7 or more and its mean elbow error drops from 19.48° to 17.32°: the filter dropped 68 people it read worse. The 254 people both settings read were read alike but for one, and on them the difference, every detection minus filtered, is -0.06° [-0.20 to +0.00]. Models given as --pred a=a.json --pred b=b.json are compared the same way, on what both read.
• Pairing. People are matched to labels image by image by box overlap; points left unlabelled in the truth stay out of the count instead of counting as errors.
• Honest intervals. Readings from one image, or from one subject with --cluster, are resampled together.
• Where it goes wrong. 58% of elbows disagree with COCO's labels by 15° or more when the arm segments average under 30 px, against 26% above 60 px. Across all arms 43% disagree, and the labels' own spread alone would put 25.9 to 39.9% there (statistics).

Install

python -m pip install poseaudit

That needs only NumPy and runs every audit. python -m pip install "poseaudit[plot]" adds --plot (Matplotlib); the extra [images] adds --images, which reads YOLO image sizes from the images (Pillow), and [supervision] adds pa.from_supervision.

Quick start

git clone https://github.com/8rulerstar/poseaudit   # the demo data (or GitHub's Code > Download ZIP)
cd poseaudit/examples/coco_elbow
poseaudit audit --format coco --gt gt_200.json --pred pred_yolo11n.json --angle 5,7,9 --big-error 15 --size-bands 30,60

This prints the summary in the GIF above: the left elbow angle of yolo11n-pose against COCO's labels on 200 val2017 images. Keypoints count from 0 in COCO's order (5, 7, 9: left shoulder, elbow, wrist; 11, 13, 15: left hip, knee, ankle; each right-side point is one higher). --big-error is required: the error that counts as large, in the measure's unit. Add --report report.md for a report that explains every line, --full for every statistic and --plot panels.png for this figure (with [plot]). If poseaudit is not found, use python -m poseaudit.

For a validation paper

poseaudit audit --format coco --gt labels.json --pred results.json --angle 5,7,9 --big-error 15 --full --json figures.json --report report.md --plot panels.png

--gt takes COCO keypoint annotations and --pred a COCO results file (or, with --format yolo, YOLO label folders). Report how many were read, the bias with its limits of agreement, the mean absolute error, ICC and the large-error rate, each with its interval; docs/statistics.md words this for the demo. Frames of one video are not independent: if image names carry the clip or participant, as clip3_000123.jpg does, add --cluster "^(clip\d+)_" (its first group names the cluster), or the intervals come out far too narrow.

As a CI gate

poseaudit audit --format coco --gt labels.json --pred results.json --angle 5,7,9 --big-error 15 --json figures.json
jq -e '(.mean_abs_error_ci[1] // 1e9) <= 25 and (.big_error_rate_ci[1] // 1) <= 0.5' figures.json

It fails unless the 95% intervals end at or below 25° of mean |error| and 50% of large errors. Resampling is seeded, so reruns agree; one image or --cluster group gives null, which fails.

How large must a limb be

poseaudit budget --format coco --gt labels.json --pred results.json --angle 5,7,9 --big-error 15 gives the smallest limb, in pixels, whose angle is off by 15° or more in at most 20% of readings, for that model on images like those, after checking its fit on readings it did not see (to within at least 6 percentage points, so a size can miss the target by that much). It needs about 140 readings of the joint, or 140 subjects or clips with --cluster, in each of its two groups (straight and bent limbs): on the 200-image demo it gives no size and says why. On COCO val2017, over four YOLO11 models at these settings, the check passed all 16 elbow groups, 11 of which gave a size (the others never get below 20%), and 3 of 16 knee groups; one elbow group passed in only 10 of 20 redraws of the data (--redraw) (budget).

Caveats

• The figures are agreement with one human label, not error against the world (demo).
• 2D angles read off an image are not 3D joint angles.
• Name the subject (subject column, --cluster), or every frame counts as independent.
• The jitter check in --full is experimental and assumes labels much cleaner than the model.

What is new versus OKS, PCK and pycocotools

They score where the points land, yet points scattered by the label spread OKS is built on turn these elbow angles by 11 to 16° on average. poseaudit scores the angle itself, with the statistics clinicians use to compare two measuring devices, from COCO, YOLO or supervision.

More

docs/README.md indexes the docs: every option, comparing models, the Python API, the statistics, the JSON output, the limitations and the CHANGELOG.

Code: MIT. Demo data: a subset of COCO's 2017 keypoint labels (© COCO Consortium, CC BY 4.0) and predictions from Ultralytics yolo11n-pose (AGPL-3.0), which poseaudit does not depend on or include.