isat-sam 1.0.0

Interactive semi-automatic annotation tool for image segmentation based on SAM(segment anything model).

ISAT_with_segment_anything

An Interactive Semi-Automatic Annotation Tool Based on Segment Anything

[中文] [English]

Our tool enables interactive use of segment anything for rapid image segmentation with low RAM requirements (optional bf16 mode).

Demo Video：YouTube

---

Features

Annotaion modes

• Semi-automatic Annotation: utilizes SAM with point and bbox prompts.
• Manual Annotation: click or drag to draw polygons (0.15s per point).

Annotation adjustments

• Polygon Adjustments: delete points and adjust object occlusions.
• Polygon Visualization: Preview groups and semantic/instance segmentation masks.

Export annotations

• Supported formats: MSCOCO, YOLO, LabelMe, and VOC (also xml)

For more features, see the Features Description.

---

Installation

There are three ways to install ISAT-SAM:

1. from source code (recommended)
2. pip install
3. from .exe

Option 1: from source code

(1) Create environment

conda create -n isat_env python=3.8
conda activate isat_env

(2) Install ISAT_with_segment_anything and its dependencies

To use GPU, please install Pytorch-GPU on Windows OS frist.

git clone https://github.com/yatengLG/ISAT_with_segment_anything.git
cd ISAT_with_segment_anything
pip install -r requirements.txt

(3) Download Segment anything pretrained checkpoint.

Download the checkpoint, and save in under: ISAT_with_segment_anything/ISAT/checkpoints

After version 0.0.3, you can manage checkpoints within GUI, click [menubar]-[SAM]-[Model manage] to open the GUI.

Now support SAM, sam-hq, MobileSAM, and EdgeSAM etc.

	pretrained checkpoint	memory	size
SAM	sam_vit_h_4b8939.pth	7305M	2.6G
	sam_vit_l_0b3195.pth	5855M	2.6G
	sam_vit_b_01ec64.pth	4149M	375M
sam-hq	sam_hq_vit_h.pth	7393M	2.6G
	sam_hq_vit_l.pth	5939M	1.3G
	sam_hq_vit_b.pth	4207M	379M
	sam_hq_vit_tiny.pth	1463M	43M
mobile-sam	mobile_sam.pt	1375M	40M
edge-sam	edge_sam.pth	960M	39M
	edge_sam_3x.pth	960M	39M
sam-med	sam-med2d_b.pth	1500M	2.4G

(4) Run

python main.py

Option 2: pip install

Note that the version may be lower than source code version if installed with pip

(1) Create environment

conda create -n isat_env python=3.8
conda activate isat_env

(2) pip install isat-sam

To use GPU, please install Pytorch-GPU on Windows OS frist.

pip install isat-sam

(3) Run

isat-sam

Option 3: install with .exe

(1) download the .exe

The version of exe maybe older than source code.

Download three .zip files, total 2.7G

	Download link
Baidu Netdisk	link：https://pan.baidu.com/s/1vD19PzvIT1QAJrAkSVFfhg code：ISAT

Click main.exe to run the tool.

(2) Download Segment anything pretrained checkpoint.

The download zip files, container sam_hq_vit_tiny.pth, but somebody told the model not support cpu. You can download mobile_sam.pt to test the tool.

If you want use other models, seeDownload Segment anything pretrained checkpoint

---

Usage

1. Annotation

1. Choose the categories in left window of software.
    Edit the category in Toolbar-File-Setting.
    
2. Start annotating
    2.1 semi-automatic annotate with SAM.
        Click button named [Segment anything] start annotate(shortcut Q).
        Click interested area with left button of mouse, click uninterested area with right button of mouse, SAM will calcute masks.
    2.2 draw polygons manually.
        Click the button [Draw polygon] to start annotation (shortcut C).
        Left click to add point into the polygon.
        Hold the left click and drag will automaticly add point into the polygon (time interval of 0.15 seconds).
    2.3 Undo
        Click the button [Backspace] to return to the previous state (shortcut Z).
3. Finish the annotation with [Annotate finished] or shortcut E.
4. Save the annotation with [Save] or shortcut S

2. Polygon Modification

1. Modify polygons coordinates
    Drag and drop polygon vertices to modify the shape of the polygon.
    Drag the polygon to adjust its position.
2. Modify polygons category
    Choose the polygon and click [Edit] or double click the polygon, and choose the new category in editing window. 
3. Occlusion modification
    Choose the polygon and click [To top] (shortcut T) or [To bottom] (shortcut B).
4. Delete polygon
    Choose the polygon and click [Delete] to delete the polygon.

3. Visualization

1. Preview annotations
    Click the [Bit map] to preview semantic and instance annotation masks (shortcut space).
    The order of swithing is polygons-semantic-instance.
2. Image window
    Click [Zoom in], [Zoom out], [Fit window] (shortcut F) to adjust the zooming distances.
3. Show / hide polygons
    Click [Visible] to show / hide polygons (shortcut V).
4. Mask aplha (only effective when using SAM)
    Drag the [mask aplha] bar to adjust the mask transparency.

4. Convet annotations

ISAT has a specific format with .json. You can use export it to other formarts.

1. ISAT to VOC
    Convert ISAT jsons to PNG images.
2. ISAT to COCO
    Convert ISAT jsons to COCO json.
3. ISAT to LABELME
    Convert ISAT jsons to LABELME jsons.
4. COCO to ISAT
    Convert COCO json to ISAT jsons.

---

Star History

Please support us with a star—it's like a virtual coffee!

Contributors

		...
yatengLG	Alias-z	...

Citation

@misc{ISAT with segment anything,
  title={{ISAT with segment anything}: An Interactive Semi-Automatic Annotation Tool Based on Segment Anything},
  url={https://github.com/yatengLG/ISAT_with_segment_anything},
  note={Open source software available from https://github.com/yatengLG/ISAT_with_segment_anything},
  author={yatengLG, Alias-z and horffmanwang},
  year={2023},
}

更新日志

[0.0.1]

• 发布测试版，版本号0.0.1

• 更新下载功能（后续需优化下载链接）

• 支持多选多边形，现在可以批量删除了

  按下Ctrl点击多边形；按下Ctrl点击右侧标注栏；按下Shift点击右侧标注栏；点击右侧标注栏，Ctrl+A全选。

• 添加转换脚本 - utils目录下

  现支持 ISAT <-> COCO, ISAT <-> YOLO, ISAT -> XML(目标检测)

• 添加了对segment-anything-fast的支持

  现支持SAM系列模型，sam-hq以及mobile-sam等后续更新

  sam-fast要求torch版本不低于2.1.1;低于版本要求时，默认导入sam

  sam_vit_h_4b8939.pth encode计算时间缩短大概4倍，显存占用缩小到5.6G

	sam	sam-fast
0	0.698307991027832	0.19336390495300293
1	0.7048919200897217	0.21175742149353027
2	0.766636848449707	0.2573261260986328
3	0.8198366165161133	0.22284531593322754

• 添加了饱和度调整

  通过拖动工具栏中的饱和度调整条，对图片进行饱和度调整。（只与显示有关，不影响sam）

• 添加track模式

  在"自动auto"和"手动manual"模式外，为组模式中添加了"跟踪track"模式。该模式下，使用[TAB]或者[`]切换目标时，组id会显示为设置为当前多边形的组id。

[0.0.2]

• 添加了模型管理界面

  现在可以方便的管理与使用模型了。

  由于sam-hq以及mobile-sam的权重链接，需要科学上网才可以访问。这两类模型下载时会经常失败 有推荐的比较好用的大文件托管服务，可以联系我

• 整合了数据转换界面并提供了新功能

  • COCO <-> ISAT
  • YOLO <-> ISAT
  • LABELME <-> ISAT
  • ISAT -> VOC(png单通道图)
  • ISAT -> VOC for object detection(xml目标检测)

• 添加了linux系统对segment-anything-fast的支持

  该功能可以保持sam分割效果的情况下，减少显存占用并提升分割速度。（目前只对sam系列模型有效）

  由于windows下需需torch版本为2.2.0+dev且需要安装较多其他依赖，因而暂时关闭了windows系统下对该功能的支持。

• 修复了遗留bug

  • 修复了转VOC后第一行一直为0的问题
  • 轮廓保存模式中-只保存最大轮廓现在严格保存面积最大的轮廓(之前使用顶点数量进行粗估计)

• 添加了对EdgeSAM的支持。

• 修复转coco后，类别ID从0开始的问题。（现在第一类的类别id为1）

• 修复sam标注过程中，切换图片文件夹后，闪退的问题

• 添加了模型国内下载链接

• 减少模型显存占用

  使用bfloat16模型后，显存需求降低，特征计算时间略微增加，最终分割效果无显著变化。

checkpoint	mem(float)	mem(bfloat16)	cost(float)	cost(bfloat16)
edge_sam.pth	360M	304M	0.0212	0.0239
edge_sam_3x.pth	360M	304M	0.0212	0.0239
mobile_sam.pt	534M	390M	0.0200	0.0206
sam_hq_vit_tiny.pth	598M	392M	0.0196	0.0210
sam_hq_vit_b.pth	3304M	1762M	0.1496	0.1676
sam_hq_vit_l.pth	5016M	2634M	0.3766	0.4854
sam_hq_vit_h.pth	6464M	3378M	0.6764	0.9282
sam_vit_b_01ec64.pth	3302M	1760M	0.1539	0.1696
sam_vit_l_0b3195.pth	5016M	2634M	0.3776	0.4833
sam_vit_h_4b8939.pth	6462M	3378M	0.6863	0.9288

0.0.3

• 更新了项目结构
• 填加了sam_med2d模型，实现对医疗数据更好的分割效果

0.0.4

• 更新了基于bounding box的自动分割功能

基于标注完成的VOC格式目标检测数据，使用标注框进行sam框提示，自动分割图像并存储为ISAT格式json。