boxmot 10.0.2

SOTA tracking methods for detection, segmentation and pose estimation models.

Real-time multi-object, segmentation and pose tracking using Yolov8 with DeepOCSORT and LightMBN

Introduction

This repository contains a highly configurable two-stage-tracker that adjusts to different deployment scenarios. It can jointly perform multiple object tracking and instance segmentation (MOTS). The detections generated by YOLOv8, a family of object detection architectures and models pretrained on the COCO dataset, are passed to the tracker of your choice. Supported ones at the moment are: DeepOCSORT LightMBN, BoTSORT LightMBN, StrongSORT LightMBN, OCSORT and ByteTrack. They can track any object that your Yolov8 model was trained to detect.

Why using this tracking toolbox?

Everything is designed with simplicity and flexibility in mind. We don't hyperfocus on results on a single dataset, we prioritize real-world results. If you don't get good tracking results on your custom dataset with the out-of-the-box tracker configurations, use the evolve.py script for tracker hyperparameter tuning.

Installation

pip install boxmot

Custom model usage

from boxmot import DeepOCSORT

tracker = DeepOCSORT()
cap = cv.VideoCapture(0)
while True:
    ret, im = cap.read()
    ...
    # dets: 
    #  - your model's nms:ed outputs of shape Nx6 (x, y, x, y, conf, cls)
    # im:
    #  - the original image (for better ReID results)
    #  - the downscaled one fed to you model
    tracker_outputs = tracker.update(dets.cpu(), im)  # --> (x, y, x, y, id, conf, cls)
    ...

Tutorials

• Yolov8 training (link to external repository) 

• Deep appearance descriptor training (link to external repository) 

• ReID model export to ONNX, OpenVINO, TensorRT and TorchScript 

• Evaluation on custom tracking dataset 

• ReID inference acceleration with Nebullvm 

Experiments

In inverse chronological order:

• Evaluation of the params evolved for first half of MOT17 on the complete MOT17

• Segmentation model vs object detetion model on MOT metrics

• Effect of masking objects before feature extraction

• conf-thres vs HOTA, MOTA and IDF1

• Effect of KF updates ahead for tracks with no associations on MOT17

• Effect of full images vs 1280 input to StrongSORT on MOT17

• Effect of different OSNet architectures on MOT16

• Yolov5 StrongSORT vs BoTSORT vs OCSORT

  • Yolov5 BoTSORT branch: https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/tree/botsort

• Yolov5 StrongSORT OSNet vs other trackers MOT17 

• StrongSORT MOT16 ablation study 

• Yolov5 StrongSORT OSNet vs other trackers MOT16 (deprecated) 

Custom object detection architecture

The trackers provided in this repo can be used with other object detectors than Yolov8. Make sure that the input to the trackers is of the following format:

Nx6 (x, y, x, y, conf, cls)

Tracking with Yolov8

$ python track.py --yolo-model yolov8n.pt      # bboxes only
                                 yolov8n-seg.pt  # bboxes + segmentation masks
                                 yolov8n-pose.pt # bboxes + pose estimation

Tracking methods

$ python track.py --tracking-method deepocsort
                                    strongsort
                                    ocsort
                                    bytetrack
                                    botsort

Tracking sources

Tracking can be run on most video formats

$ python track.py --source 0                               # webcam
                           img.jpg                         # image
                           vid.mp4                         # video
                           path/                           # directory
                           path/*.jpg                      # glob
                           'https://youtu.be/Zgi9g1ksQHc'  # YouTube
                           'rtsp://example.com/media.mp4'  # RTSP, RTMP, HTTP stream

Select Yolov8 model

There is a clear trade-off between model inference speed and overall performance. In order to make it possible to fulfill your inference speed/accuracy needs you can select a Yolov5 family model for automatic download. These model can be further optimized for you needs by the export.py script

$ python track.py --source 0 --yolo-model yolov8n.pt --img 640
                                          yolov8s.tflite
                                          yolov8m.pt
                                          yolov8l.onnx 
                                          yolov8x.pt --img 1280
                                          ...

Select ReID model

Some tracking methods combine appearance description and motion in the process of tracking. For those which use appearance, you can choose a ReID model based on your needs from this ReID model zoo. These model can be further optimized for you needs by the reid_export.py script

$ python track.py --source 0 --reid-model lmbn_n_cuhk03_d.pt
                                          osnet_x0_25_market1501.pt
                                          mobilenetv2_x1_4_msmt17.engine
                                          resnet50_msmt17.onnx
                                          osnet_x1_0_msmt17.pt
                                          ...

Filter tracked classes

By default the tracker tracks all MS COCO classes.

If you want to track a subset of the classes that you model predicts, add their corresponding index after the classes flag,

python track.py --source 0 --yolo-model yolov8s.pt --classes 16 17  # COCO yolov8 model. Track cats and dogs, only

Here is a list of all the possible objects that a Yolov8 model trained on MS COCO can detect. Notice that the indexing for the classes in this repo starts at zero

MOT compliant results

Can be saved to your experiment folder runs/track/<yolo_model>_<deep_sort_model>/ by

python track.py --source ... --save-txt

Tracker hyperparameter tuning

We use a fast and elitist multiobjective genetic algorithm for tracker hyperparameter tuning. By default the objectives are: HOTA, MOTA, IDF1. Run it by

$ python evolve.py --tracking-method strongsort --benchmark MOT17 --n-trials 100  # tune strongsort for MOT17
                   --tracking-method ocsort     --benchmark <your-custom-dataset> --objective HOTA # tune ocsort for maximizing HOTA on your custom tracking dataset

The set of hyperparameters leading to the best HOTA result are written to the tracker's config file.

Contact

For Yolov8 tracking bugs and feature requests please visit GitHub Issues. For business inquiries or professional support requests please send an email to: [e-mail]