Open vocabulary segmentation

Models

• GroundingDINO
• MobileSAMv2:
  • trt (predictor, generator)
  • torch (predictor)
• MobileSAMv1 (predictor, generator)

Usage

Get detection

from ovsegmentation import build_segmentator, visualize, get_waypoint, OVSegmentation
input_data = {
    "image": image,
    "text": ovd_prompt,
    "depth": depth_image,
    "intrinsics": intrinsics,
    "tf": None 
}
Note: "tf" could be None or transform_matrix: np.array (4,4) 

ovs = build_segmentator(weights_path='weights')

res = ovs.get_detections(input_data, detect_most_confident=False)
# res = ovs.generate_masks(input_data)

Visualize

annotated_image, labels = visualize(
    image, res, instance_random_color=True, draw_bbox=True
)
plt.figure(figsize=(15, 8))
plt.imshow(annotated_image)
plt.show()

Weights structure

Weights structure:

• MobileSAMv2:
  • trt
    • WEIGHTS_DIR/MobileSAM/l2_encoder.engine
    • WEIGHTS_DIR/MobileSAM/mobilesamv2_decoder.engine
    • WEIGHTS_DIR/MobileSAM/ObjectAwareModel.engine
  • torch
    • WEIGHTS_DIR/MobileSAM/Prompt_guided_Mask_Decoder.pt
    • WEIGHTS_DIR/MobileSAM/l2.pt
• MobileSAMv1:
  • WEIGHTS_DIR/MobileSAM/mobile_sam.pt
• GroundingDINO:
  • WEIGHTS_DIR/GroundingDINO/groundingdino_swint_ogc.pth

weights_path: WEIGHTS_DIR