castle-ai 0.0.16

Distinguish behavioral clusters Toolbox

CASTLE (Combined Approach for Segmentation and Tracking with Latent Extraction) is a training-free framework that combines segmentation models, tracking algorithms, and visual foundation models to automatically discover animal behaviors from video. Through focused latent extraction and hierarchical clustering, it achieves expert-level accuracy across multiple species without manual labeling, while uncovering previously hidden behavioral patterns that keypoint methods miss.

Latest updates

• 2024-09: Public release of this tool.

Quick start

Option 1 (Open In Colab (free accounts are vary slow))

Option 2 (Docker)

Option 3 (Directly Install Dependency)

Installation

git clone https://github.com/CASTLE-ai/castle-ai.git
cd castle-ai
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Sometime the ckpt download may be blocked by Google. So you can download the models from the web by copying the links to the Chrome browser and downloading them.

https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth
https://dl.fbaipublicfiles.com/dinov2/dinov2_vitb14/dinov2_vitb14_reg4_pretrain.pth
https://drive.google.com/file/d/1g4E-F0RPOx9Nd6J7tU9AE1TjsouL4oZq/edit
https://drive.google.com/file/d/1QoChMkTVxdYZ_eBlZhK2acq9KMQZccPJ/edit

Afterward, place them into the ckpt folder.

castle-ai
├── castle
└── ckpt
    ├── dinov2_vitb14_reg4_pretrain.pth
    ├── R50_DeAOTL_PRE_YTB_DAV.pth
    ├── sam_vit_b_01ec64.pth
    └── SwinB_DeAOTL_PRE_YTB_DAV.pth

My Python version is 3.10.12. For details version, see INSTALLATION.md.

Run

python app.py

About us

CASTLE is a project by the Wu Lab, a research group at the Academia Sinica.

Credits & Licenses

This project incorporates code and methodologies from the following sources:

• SAM (Segment Anything Model): https://github.com/facebookresearch/segment-anything (Apache License 2.0)
• DeAOT (Decoupling Features in Hierarchical Propagation): https://github.com/yoxu515/aot-benchmark (BSD 3-Clause License)
• DINOv2 (Self-Supervised Vision Transformer): https://github.com/facebookresearch/dinov2 (Apache License 2.0)

This work is distributed under the terms of the Apache License 2.0.

Citation

If you find this work useful, please consider citing:

@article{CASTLE,
  title={CASTLE: a training‑free foundation‑model pipeline for unsupervised, cross‑species behavioral classification},
  author={Liu, Yu-Shun and Yeh, Han-Yuan and Hu, Yu-Ting and Wu, Bing-Shiuan and Chen, Yi-Fang and Yang, Jia-Bin and Jasmin, Sureka and Hsu, Ching-Lung and Lin, Suewei and Chen, Chun-Hao and Wu, Yu-Wei},
  journal={bioRxiv},
  year={2025}
}