D-FINE-seg

Real-Time Object Detection, Instance and Semantic Segmentation

Quick Start • Usage • Export • Inference • Benchmarks • Video Tutorial • Colab

D-FINE-seg is a framework for real-time object detection, instance segmentation, and semantic segmentation - one codebase, one config flag (task: detect | segment | sem_seg), five model sizes (N -> X).

• End-to-end workflow - dataset prep -> training (DDP, EMA, AMP, mosaic) -> export (ONNX, TensorRT, OpenVINO, CoreML, LiteRT) -> benchmarked multi-backend inference
• Accuracy - Cityscapes semantic segmentation at native 1024x2048: D-FINE-X reaches 84.84 mIoU at 14.6 ms end-to-end, the highest of 28 models - +1.26 mIoU over YOLO26-x, and +1.5 mIoU over Mask2Former Swin-L with 6.4x lower latency and a quarter of the params - and D-FINE-L matches or beats YOLO26-x on every axis; also beats YOLO26 and RF-DETR on 640 Cityscapes detection & instance-seg F1, and YOLO26 on TACO and VisDrone (TensorRT FP16, end-to-end protocol)
• Paper - D-FINE-seg: Object Detection and Instance Segmentation Framework with Multi-Backend Deployment
• Not a fork: the detection core follows the D-FINE paper; segmentation heads, training, export and inference are implemented from scratch.

One frame, three tasks, one config flag:

_{Full tables below}

Highlights

• Instance segmentation head (task: segment) - lightweight mask head on top of D-FINE's HybridEncoder PAN outputs: stride 8/16/32 features fused to 1/4 resolution, then a dot-product between per-query mask embeddings (3-layer MLP) and the shared mask features yields per-instance masks
• Semantic segmentation head (task: sem_seg) - reuses the pretrained instance-seg mask fuser on full-frame features, followed by a small conv neck and 1x1 classifier: no queries, no NMS
• Mask-aware training - box-cropped BCE + Dice mask losses (instance seg) and CE + multi-class soft Dice with ignore_index (semantic seg), mask supervision inside contrastive denoising, and Dice + sigmoid-focal mask costs in the Hungarian matcher - all train-time only, zero inference cost
• Pretrained weights for every task - detection on COCO (plain and Objects365-pretrained), COCO instance segmentation, Cityscapes semantic segmentation
• Multi-channel inputs - train on RGB + thermal / depth / NIR stacks (4-channel .npy), not just RGB
• Modern training stack - Muon optimizer, DDP, EMA, mosaic + affine augs, knowledge distillation, OneCycle, early stopping, rare-class sampling (LVIS repeat-factor), WandB, auto-batch size.
• Beyond the model - ByteTrack tracking, SAM3 auto-labeling, Gradio demo, INT8 quantization (OpenVINO / CoreML / LiteRT)

Quick Start

Installation

pip install dfine-seg           # inference + training
pip install 'dfine-seg[all]'    # + every export backend, SAM3, Gradio demo

Pretrained weights auto-download from Hugging Face on first use: detection on COCO (dfine_{n,s,m,l,x}_coco.pt, and dfine_{n,s,m,l,x}_obj2coco.pt - Objects365 pretraining, then COCO), COCO instance segmentation (dfine_seg_{n,s,m,l,x}_coco.pt), Cityscapes semantic segmentation (dfine_sem_{n,s,m,l,x}_cityscapes.pt).

Two-line predict:

from dfine_seg import load_model, read_image, Visualizer

model = load_model("s")                              # COCO detection, weights auto-downloaded
model = load_model("s", task="segment")              # COCO instance segmentation
model = load_model("x", task="sem_seg")              # Cityscapes semantic segmentation
model = load_model("output/models/exp/model.pt")     # your checkpoint - size/task/classes/input size auto-detected, supports: .pt | .engine | .onnx | .xml

img = read_image("path/to/image.jpg")
out = model(img)[0]
print(out["boxes"], out["scores"], [model.names[int(i)] for i in out["labels"]])

drawn = Visualizer(model)(img, out)                  # annotated BGR copy - boxes, masks or a sem_seg overlay

load_model returns the same wrapper you would construct by hand (dfine_seg/infer/) - it resolves the weights and picks the backend. Extra keyword arguments pass straight through (load_model("s", conf_thresh=0.3)), output tensors stay on the device the model ran on, and those wrapper files remain self-contained enough to copy into your own app. Visualizer reads the class count and names off the model it is given, then draws whatever that model returned - boxes, instance masks or a dense label map - so one call covers every task (BGR uint8 in, BGR uint8 out).

To train from a pip install, materialize a config and go:

dfine init          # writes ./config.yaml - edit train.root and train.label_to_name
dfine split
dfine train         # Hydra overrides work: dfine train model_name=m train.epochs=100

From source (contributors)

git clone https://github.com/ArgoHA/D-FINE-seg.git
cd D-FINE-seg
uv sync

Prepare Your Data

Two annotation formats are supported: YOLO (default) and COCO JSON. Semantic segmentation uses PNG masks instead (see below).

YOLO format (default)

data/dataset/
├── images/    # all images: .jpg, .png, etc. (.npy for multi-channel - see below)
└── labels/    # all labels: one .txt per image (same filename stem)

Detection labels: class_id xc yc w h (normalized)

Segmentation labels: class_id x1 y1 x2 y2 ... xN yN (normalized polygon coordinates)

Input types & channel order: 3-channel .jpg/.png (BGR, read via cv2.imread), 3-channel .npy (RGB, read via np.load), or 4-channel .npy (RGB+extras, e.g. RGB+thermal).

Semantic segmentation masks (task: sem_seg)

data/dataset/
├── images/    # same as YOLO layout
└── labels/    # one single-channel uint8 .png per image (same stem), pixel value = class id

Every pixel gets a class from label_to_name (background included). Pixels with value train.sem_seg.ignore_index (default 255) are excluded from loss and metrics and during inference 255 is the "background" or "ignored" class. dfine split works unchanged; keep_ratio: True is supported (letterbox pad is filled with ignore_index, so pad pixels don't supervise); coco_dataset: True is not supported for this task.

Released Cityscapes weights load with load_model("x", task="sem_seg") - names and input size come from the checkpoint. They are also a strong init for your own driving/outdoor data: point train.pretrained_model_path at one with a different label_to_name and the class heads re-initialize while the rest keeps the Cityscapes training (N from COCO detection weights, S/M/L/X from obj2coco). A model fine-tuned from them is still derived from Cityscapes and inherits its non-commercial terms (see License) - for commercial use, start from the COCO / obj2coco weights instead. Benchmark numbers and the full 28-model comparison: cityscapes-benchmark.

Multi-channel inputs (RGB + thermal / depth / NIR / …)

Set train.in_channels: 4 (3 or 4 supported) to train on RGB + one extra modality (thermal / depth / NIR). Stacks are .npy uint8 HWC arrays in images/ (RGB in planes 0-2, extras after) with YOLO labels as usual; a mismatched channel count is skipped with a warning. See dfine_seg/etl/m3fd_to_yolo.py for a ready-made RGB+thermal converter.

COCO JSON format

Place standard COCO JSON annotation files alongside your images folder. Splits are detected automatically by filename:

data/dataset/
├── images/       # all images
├── train.json    # COCO-format annotations for train split
├── val.json      # COCO-format annotations for val split
└── test.json     # (optional) COCO-format annotations for test split

Enable COCO mode by setting coco_dataset: True in your config (see below). No CSV split generation step is needed - the splits are read directly from the JSON files.

If you only have a single coco.json, run dfine split to produce train.json / val.json (and test.json when the ratios leave room) from it. It splits by image using the same split: ratios, seed and ignore_negatives as the YOLO path, keeps each image's annotations with it, and copies categories / info / licenses into every output.

Configure

Edit config.yaml - key settings:

task: detect  # detect | segment | sem_seg
exp_name: my_exp  # experiment name (used in output paths)
model_name: s  # n / s / m / l / x

train:
  root: /path/to/project  # project root, will be used for outputs
  data_path: /path/to/dataset  # folder with images/ and labels/ (YOLO) or *.json files (COCO)
  coco_dataset: False  # set True to use COCO JSON annotations (train.json / val.json / test.json)
  label_to_name:
    0: class_a
    1: class_b
  epochs: 75
  batch_size: 8
  img_size: [640, 640]  # (h, w)

Usage

Notes:

• Most commands read config.yaml and work off your trained run; exceptions — init (writes the config), predict, hw_bench, demo, version (no config needed). Every command's flags, defaults and examples: dfine <command> -h.
• YOLO format: dfine train requires train.csv and val.csv in train.data_path (generated by dfine split).
• COCO format: set coco_dataset: True - train.json and val.json are loaded directly; dfine split is only needed if you have a single coco.json to split.
• dfine infer runs Torch inference on train.path_to_test_data and writes to train.infer_path. infer / export / bench auto-pick the latest <exp_name>_<date> run under train.path_to_save.

Every dfine command also has a make alias (make train == dfine train), and any config key can be overridden inline:

dfine train exp_name=my_exp model_name=m train.epochs=100

Enable DDP (multi-GPU) by setting train.ddp.enabled: True and train.ddp.n_gpus: N in config. Then just run dfine train - it auto-launches with torchrun.

Training Features

Export

Tip: FP16 is the best latency/accuracy trade-off for GPU (TensorRT) and CPU (OpenVINO). For Apple Silicon (CoreML), FP32 is faster.

Warning: run TensorRT engines at batch 1. On TRT 10.13.3.9 a batched engine does not compute batch elements independently - feeding four identical images through a single execute_async_v3 call returns four different results (score spread up to 0.20, and different labels), so a detection's score depends on what it happened to be batched with. Batch 1 is exact, and reproduces in raw TensorRT for FP16 and FP32 and for every optimization-profile shape, while torch and ONNX Runtime stay identical across slots (NVIDIA/TensorRT#4813).

After export, a parity self-check (export.parity, on by default) runs each backend on a shared input and writes one cosine per backend - over the sorted top-K detection scores vs torch - to parity.csv next to the weights.

For task: sem_seg every backend gets the same fused-argmax graph: a single int32 sem_seg output [B, H, W] (label map at input resolution, no detection postprocessor), and parity compares per-pixel argmax agreement instead of score cosine.

Inference

Backends

Six inference backends in dfine_seg/infer/:

Output contract: detection / instance segmentation wrappers return labels, boxes, scores (+ masks [N, H, W] for segment); sem_seg wrappers return a single sem_seg [H, W] label map at original image resolution. For sem_seg, dfine infer writes palette overlays + GT-style grayscale PNG label maps (crops and tracking are box-based and skipped).

Also provided:

• Bytetrack - simple implementation of object tracker
• SAM3 - text-promptable zero-shot segmentation for auto-labeling (multi-class: repeat --prompt or pass "car, person")

Multi-Object Tracking

A simplified ByteTrack (Zhang et al., ECCV 2022) is included for persistent object tracking across video frames - uses constant-velocity motion prediction with EMA-smoothed velocity instead of a Kalman filter, blends IoU with centroid distance in the match cost, and does per-class matching by default.

Gradio Demo

dfine demo         # == make demo   (pip: pip install 'dfine-seg[demo]')

A web UI for running inference on uploaded images and videos (or a webcam snapshot). It starts on COCO detection s - no setup. The Model panel then swaps in any other model at runtime: a size preset, or a path/upload of your own .pt / .onnx / .engine / .xml, with a box for the class names. detect, segment and sem_seg checkpoints all render, and SAM3 is selectable as a second backend for text-promptable segmentation (prompts are comma- or newline-separated - each one becomes a class).

It serves on 0.0.0.0:7860 (LAN-reachable) and prints a warning on startup: the Model panel loads any path the browser sends, so anyone who can reach the port can load files off this machine. dfine demo --host 127.0.0.1 restores local-only.

Benchmarks

Semantic segmentation - Cityscapes at native 1024x2048

28 models from 5 families: the five D-FINE-seg sizes trained by us and 23 official Cityscapes checkpoints (YOLO26-sem, EfficientViT-Seg, SegFormer, Mask2Former), none of them retrained. Input is the whole 1024x2048 frame in one forward pass; each model runs its own framework's shipped pre- and postprocessing and is scored by one validator on the 500 val images. TensorRT fp16, batch 1, RTX 5070 Ti. e2e ms includes each framework's CPU pre- and postprocessing; engine ms is pure TensorRT execute. Full table, per-class IoU, protocol and every known asymmetry: cityscapes-benchmark.

• D-FINE-X vs Mask2Former-Swin-L (the most accurate Mask2Former checkpoint): +1.51 mIoU, 4.5x lower engine latency, 6.4x lower end-to-end latency, about a quarter of the parameters. X has the highest mIoU and pixel accuracy of the 28 models.
• D-FINE-L vs YOLO26x-sem (the most accurate YOLO26-sem checkpoint): +0.22 mIoU, +0.14 pixel acc, 13% lower engine latency, 28% lower end-to-end latency, 38% fewer parameters. One training seed and a small margin, so read it as "matches or beats".

Cityscapes - vs YOLO26 and RF-DETR (fine-tuning)

500 Cityscapes val images at original 2048x1024, TensorRT 10.13 FP16, batch 1, RTX 5070 Ti - same protocol and latency columns as above, with confidence thresholds tuned per framework to maximize F1. Full protocol: cityscapes-benchmark.

Detection

Instance segmentation

The 640 semantic segmentation comparison is superseded by the native-resolution results above. Metric definitions, the VisDrone and TACO benchmarks, COCO-style APs and the export-format comparisons: docs/benchmarks.md

Outputs

Result examples

Training

Benchmarking

WandB dashboard

Inference

Citation

If you use D-FINE-seg in your research, please cite:

@article{saakyan2026dfineseg,
  title={D-FINE-seg: Object Detection and Instance Segmentation Framework with multi-backend deployment},
  author={Saakyan Argo and Solntsev Dmitry},
  eprint={2602.23043},
  journal={arXiv preprint arXiv:2602.23043},
  year={2026}
}

And the original D-FINE paper:

@misc{peng2024dfine,
      title={D-FINE: Redefine Regression Task in DETRs as Fine-grained Distribution Refinement},
      author={Yansong Peng and Hebei Li and Peixi Wu and Yueyi Zhang and Xiaoyan Sun and Feng Wu},
      year={2024},
      eprint={2410.13842},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

If you use the Cityscapes-trained semantic-segmentation weights, also cite the Cityscapes dataset:

@inproceedings{Cordts2016Cityscapes,
  title={The Cityscapes Dataset for Semantic Urban Scene Understanding},
  author={Cordts, Marius and Omran, Mohamed and Ramos, Sebastian and Rehfeld, Timo and Enzweiler, Markus and Benenson, Rodrigo and Franke, Uwe and Roth, Stefan and Schiele, Bernt},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2016}
}

License

The code is licensed under the Apache 2.0 License.

The released Cityscapes semantic-segmentation weights (dfine_sem_{n,s,m,l,x}_cityscapes.pt) are derivative works of the Cityscapes dataset and inherit its terms of use: they may be distributed for non-commercial purposes such as academic research, teaching, scientific publications, or personal experimentation, but not for commercial gain. Train on your own data and the resulting model is yours to ship commercially - the code stays Apache-2.0 (no AGPL).

Acknowledgement

The detection core is based on the D-FINE paper and architecture. The mask head design follows the Mask DINO paradigm. Thank you to both teams for their excellent work.

Benchmarks in this project use the Cityscapes, VisDrone and TACO datasets. We thank the authors for making these datasets publicly available.