project_description
Project description
simple-cocotools
A simple, modern alternative to pycocotools
.
About
Why not just use Pycocotools?
- Code is more readable and hackable.
- Metrics are more transparent and understandable.
- Evaluation is fast.
- Only dependencies are
numpy
andscipy
. Nocython
extensions. - Code is more modern (type annotations, linting, etc).
Install
From PyPI
pip install simple-cocotools
From Repo
pip install "simple-cocotools @ git+ssh://git@github.com/fkodom/simple-cocotools.git"
For Contributors
# Clone this repository
gh repo clone fkodom/simple-cocotools
cd simple-cocotools
# Install all dev dependencies (tests etc.)
pip install -e .[all]
# Setup pre-commit hooks
pre-commit install
Usage
Expects target annotations to have the same format as model predictions. (The format used by all torchvision
detection models.) You may already have code to convert annotations into this format, since it's required to train many detection models. If not, use 'AnnotationsToDetectionFormat' from this repo as an example for how to do that.
A minimal example:
from torchvision.detection.models import maskrcnn_resnet50_fpn
from simple_cocotools import CocoEvaluator
evaluator = CocoEvaluator()
model = maskrcnn_resnet50_fpn(pretrained=True).eval()
for images, targets in data_loader:
predictions = model(images)
evaluator.update(predictions, targets)
metrics = evaluator.summarize()
metrics
will be a dictionary with format:
{
"box": {
"mAP": 0.40,
"mAR": 0.41,
"class_AP": {
"cat": 0.39,
"dog": 0.42,
...
},
"class_AR": {
# Same as 'class_AP' above.
}
}
"mask": {
# Same as 'box' above.
}
}
For a more complete example, see scripts/mask_rcnn_example.py
.
Benchmarks
I benchmarked against several torchvision
detection models, which have mAP scores reported on the PyTorch website.
Using a default score threshold of 0.5:
Model | Backbone | box mAP (official) |
box mAP | box mAR | mask mAP (official) |
mask mAP | mask mAR |
---|---|---|---|---|---|---|---|
Mask R-CNN | ResNet50 | 37.9 | 36.9 | 43.2 | 34.6 | 34.1 | 40.0 |
Faster R-CNN | ResNet50 | 37.0 | 36.3 | 42.0 | - | - | - |
Faster R-CNN | MobileNetV3-Large | 32.8 | 39.9 | 35.0 | - | - | - |
Notice that the mAP for MobileNetV3-Large
is artificially high, since it has a much lower mAR at that score threshold. After tuning the score threshold, so that mAP and mAR are more balanced:
Model | Backbone | Threshold | box mAP | box mAR | mask mAP | mask mAR |
---|---|---|---|---|---|---|
Mask R-CNN | ResNet50 | 0.6 | 41.1 | 41.3 | 38.2 | 38.5 |
Faster R-CNN | ResNet50 | 0.6 | 40.8 | 40.4 | - | - |
Faster R-CNN | MobileNetV3-Large | 0.425 | 36.2 | 36.2 | - | - |
These scores are more reflective of model performance, in my opinion. Mask R-CNN slightly outperforms Faster R-CNN, and there is a noticeable (but not horrible) gap between ResNet50 and MobileNetV3 backbones. PyTorch docs don't mention what score thresholds were used for each model benchmark. ¯\(ツ)/¯
Ignoring the time spent getting predictions from the model, evaluation is very fast.
- Bbox: ~400 samples/second
- Bbox + mask: ~100 samples/second
- Using a Google Cloud
n1-standard-4
VM (4 vCPUs, 16 GB RAM).
Note: Speeds are dependent on the number of detections per image, and therefore dependent on the model and score threshold.
How It Works
TODO: Blog post on how simple-cocotools
works.
- Match the predictions/labels together, maximizing the IoU between pairs with the same object class. SciPy's
linear_sum_assignment
method does most of the heavy lifting here. - For each IoU threshold, determine the number of "correct" predictions from the assignments above. Pairs with IoU < threshold are incorrect.
- For each image, count the number of total predictions, correct predictions, and ground truth labels for each object class and IoU threshold.
- Compute AP/AR for each class from the prediction counts above. Then compute mAP and mAR by averaging over all object classes.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for simple_cocotools-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 82384e362fb712ffcb604be023a63740d2996f442985b64421225a16e7da6432 |
|
MD5 | d8918e5dd0b6690f4d0e954c66177fb1 |
|
BLAKE2b-256 | 33b139e777324f461bcfebececcbc3a12c324311699b59bec37333e2b939226a |