AutoSegmentor

AutoSegmentor is a state-of-the-art auto-labeling ecosystem that bridges the gap between raw video footage and structured AI datasets. By integrating Meta AI's Segment Anything Model 2 (SAM2) with high-precision tracking like CoTracker3, it enables users to generate pixel-perfect masks and pose estimation data for long, complex videos with minimal manual interaction.

📖 Full documentation, demos, and architecture guide →

✨ Features

• Professional Desktop UI: A fully-featured PyQt5 application with multi-window support, integrated property panels, and real-time visualization.
• Interactive Annotation: Point and box-based multi-class annotation with a high-fidelity zoom system for precision.
• Advanced Tracking (CoTracker3): Robust keypoint tracking across frames — an alternative to Optical Flow for complex scenes.
• Real-time Mask Propagation: Propagate annotations across batches of frames using SAM2's temporal memory.
• Async Processing Engine: Background execution of GPU tasks keeps the UI responsive during heavy inference.
• YOLO Dataset Creation: One export covers object detection (bbox), instance segmentation, and pose estimation simultaneously, with integrated augmentation.

🚀 Quickstart

Tested on Windows 11 and Ubuntu 22.04/24.04. Full walkthrough (prerequisites, manual install path, troubleshooting): Installation guide →

git clone --recursive https://github.com/thippeswammy/AutoSegmentor.git
cd AutoSegmentor
python -m venv .venv && source .venv/bin/activate   # or .venv\Scripts\Activate.ps1 on Windows
python install.py
python run_main.py --demo cat

install.py is a single cross-platform script that installs dependencies, initializes submodules, downloads the SAM2 + CoTracker3 checkpoints, and runs a GPU diagnostic — see python install.py --help for flags to skip or isolate individual steps.

Just want the annotation engine — SAM2 masks and CoTracker3 pose tracking — without cloning the repo? pip install autosegmentor pulls both straight from PyPI. Demos and the Dataset Manager / Synthetic Engine tooling are still git-clone-only for now — planned for a future release.

🎬 Demos

python run_main.py --demo list           # cat, road
python run_main.py --demo cat
python run_main.py --demo road

See Demos → for what each bundled demo shows. The road demo (SAM2 segmentation only, no pose tracking):

Using your own video? Drop it in workspace/VideoInputs/ and just run python run_main.py (no --demo) — a Setup Dialog opens where you pick the video and configure SAM2/CoTracker3, run mode, and pose classes, then the same annotation workflow as the demos takes over. See the Installation guide → for details.

⌨️ Annotation Controls

🏗️ Architecture

A PyQt5 annotation UI drives a background engine wrapping SAM2 (mask propagation) and CoTracker3 (keypoint tracking), with a separate downstream toolchain (DatasetManager/) turning verified annotations into YOLO-format training data. For the full call flow, diagram, and package breakdown, see the Architecture guide →.

AutoSegmentor/
├── run_main.py                # Main entry point
├── install.py                 # One-shot setup (deps, submodules, checkpoints, GPU check)
├── autosegmentor/              # Core application package (core, ui, models, file_management, tools)
├── DatasetManager/              # Dataset export & synthesis — see the Dataset Manager guide
├── workspace/                  # Project workspace (videos in, datasets/logs out)
├── external/                   # Vendored SAM2 + CoTracker3
├── demo/                        # Bundled demo footage + session configs
└── docs/                        # Source for the documentation site

Acknowledgements

• Meta AI's SAM2
• CoTracker Team
• All open-source contributors to the PyTorch and PyQt ecosystems.

Built with ❤️ for the Computer Vision community.