openlabcluster 0.0.37

OpenLabCluster

OpenLabCluster

Active Learning Based Clustering and Classification of Animal Behaviors in Videos Based on Automatically Extracted Kinematic Body Keypoints

Usage

Installation

Install from pip

Create a new environment with conda and install the package

For Linux start the environment with spec-file.txt

conda create --name OpenLabCluster --file spec-file.txt
conda activate OpenLabCluster
pip install openlabcluster

for Mac OS and Windows

conda create --name OpenLabCluster python=3.7
conda activate OpenLabCluster
pip install 'openlabcluster[gui]'

Execution

Run the following for Linux

python -m openlabcluster

Run the following for Mac-OS

conda install python.app
pythonw -m openlabcluster

Run the following for Windows

pythonw.exe -m openlabcluster

Run a Demo

Create the Demo Project

1. Download the example folder from here including the extracted keypoints of Home-Cage Mouse dataset (from Automated home-cage behavioural phenotyping of mice Hueihan Jhuang et al., 2010) and follow the instructions in the README file in the example folder.

2. Go to your_download-dir/example folder run

    python3 prepare_video_list.py
   

   which generates video_segments_names.text file.

3. Launch OpenLabCluster GUI (see Execution above)

4. Set Project Name: e.g., demo

5. Click Load Preprocessed Keypoints, choose datafile: your_download_dir/openlabcluster_example/demo_data.h5

6. Click Load Video Segments Name List, choose the file: your_download_dir/openlabcluster_example/video_segments_names.text

7. Uncheck Check to use GPU, if GPU is not available.

8. Set Feature Length = 16

9. Click OK to create the project

Start the Demo Project:

1. Go to Manage Project panel,
2. Choose Load Project
3. Select the config file as filedirectory/project_name/config.yaml
4. Click OK, then go to Cluster Map

Cluster Map:

1. Click Start Clustering button to start unsupervised clustering.
2. Click Go To Classification when unsupervised clustering is finished, then go to Behavior Classification Map panel.

Behavior Classification Map:

1. The scatter plot indicating sample clusters is initialized on the bottom left, with suggested samples for annotation.
2. Label samples on the bottom right panel.
3. Click Run Classification, and start classification.

Manage Project (Start a New Project or Load Earlier Project) Detailed Description

Start a New Project

1. Project Name - the name of the project
2. If you have only videos, use markerless pose estimators (e.g., DeepLabCut) to extract keypoints for each video. If you already have DeepLabCut-like formatted files, select them with "Load Keypoints Data."
3. List video names in the "Load Video Segments Names List" file. There should be one video for each keypoint file. Make sure the videos and keypoints files are in the same order.
4. Optional: Set a directory for the project (the default is the working directory of this project).
5. Keep the GPU box checked if you have a GPU on your computer and you would like to use it for training.
6. Enter the features length (number of body parts * number of dimensions per body part. For example, 5 keypoints in 2D would be 5*2 =10).
7. Choose "OK" to create the project.
8. If you want to edit the config (i.e., change class names, change video cropping), press "Edit Config File."

Loading a Project

1. Select the config.yaml file generated when you created that project.
2. Press "OK."

Cluster Map (Unsupervised Learning)

Set the Training Parameters

1. Update Cluster Map Every (Epochs): The frequency to update the Cluster Map (bottom left panel in the figure), e.g., set 1 to update Cluster Map every training epoch, or 5 to update every 5 epochs.
2. Save Cluster Map Every (Epochs): The frequency to save the Cluster Map.
3. Maximum Epochs: The maximum number of epochs to be performed unsupervised training.
4. Cluster Map Dimension: Possible choices are 2d or 3d. For 2d, the Cluster Map will be shown in two dimensions; otherwise, it will be three dimensions.
5. Dimension Reduction Methods: Possible choices are PCA, tSNE, and UMAP. The GUI will use the chosen method to perform dimension reduction and show results in Cluster Map.

Buttons

After setting the parameters, you can perform the analysis:

1. Start Clustering: Perform an unsupervised sequence regeneration task.
2. Stop Clustering: Usually, the clustering will stop when it reaches the maximum epochs, but if you want to stop at an intermediate stage, click this button.
3. Continue Clustering: If you stopped the clustering at some stage and want to perform clustering with earlier clustering results, click this button.
4. Go to Classification: After the unsupervised clustering, we go to the next step, which includes: i) annotation suggestion, ii) sample annotating, and iii) semi-supervised action classification with labeled samples.

Behavior Classification Map

Set the Training Parameters

1. Selection Method: In this part, your selection will decide which method GUI uses to select samples for annotation. There are four possible choices: Marginal Index (MI), Core Set (CS), Cluster Center (TOP), Cluster Random (Rand), and Uniform.
2. # Samples per Selection: the number of samples you want to label in the current selection stage. You can select or deselect samples in the behavior classification map.
3. Maximum Epochs: The maximum epoch the network will be trained when performing the action recognition.
4. Cluster Map Dimension: You can choose 2d or 3d. If it is 2d, the Cluster Map will be shown in two-dimensional space; otherwise, it is three-dimensional space.
5. Dimension Reduction Method: Possible choices are PCA, tSNE, and UMAP. The GUI will use the chosen reduction method to perform dimension reduction and show results in the Cluster Map.

Buttons:

1. Run Classification: Save annotation results and train the action recognition model.

2. Stop Classification: Stop training.

3. Next Selection: Suggest a new set of samples based on with indicated active learning method. Notice: If you change Cluster Map Dimension or Dimension Reduction Method, click the Next Selection to show suggested samples.

4. Get Results: Get the Behavior Classification Map (predicted class label) from the trained model on unlabeled samples.

Plots

1. Behavior Classification Plot:

   Visualization mode: The points representing each action segment is shown in black for visualization. It can be visualized in 2D or 3D with different dimension reduction methods.

   Annotation mode: In this mode, dots for each action segment in a different color are shown in the Behavior Classification plot (only in 2D). Red: current sample for annotating, and the video segment is shown on the right. Blue: the suggested samples for annotating in this iteration (deselect them by clicking the point). Green: Samples have been annotated.

   Buttons:

   • Zoom: Zoom in or zoom out the plot.
   • Pan: Move the plot around.

2. Videos Panel:

   Left panel: The corresponding video of the action segment, which is shown in the Behavior Classification Map in red.

   Right panel: The class name and class id. According to the video, one can label the action segment.

   Buttons:

   • Previous: Load the previous video.
   • Play: Play the video.
   • Next: Go to the next video.

Attribution

OpenLabCluster code was developed in University of Washington UW NeuroAI Lab by Jingyuan Li and Moishe Keselman. OpenLabCluster interface is inspired by DeepLabCut, which code is used as a backbone for user interface panels, interaction with the back-end, logging, and visualization. For specific usage please see third_party/deeplabcut folder. OpenLabCluster also uses Google Active Learning Playground code for implementation of the K-center selection method in Core-Set active learning option. For specific usage please see third_party/kcenter folder.

Related projects and publications:

OpenLabCluster

@article{li2022openlabcluster,
  title={OpenLabCluster: Active Learning Based Clustering and Classification of Animal Behaviors in Videos Based on Automatically Extracted Kinematic Body Keypoints},
  author={Li, Jingyuan and Keselman, Moishe and Shlizerman, Eli},
  journal={bioRxiv},
  pages={2022--10},
  year={2022},
  publisher={Cold Spring Harbor Laboratory}
}

DeepLabCut

@article{Mathisetal2018,
    title = {DeepLabCut: markerless pose estimation of user-defined body parts with deep learning},
    author = {Alexander Mathis and Pranav Mamidanna and Kevin M. Cury and Taiga Abe  and Venkatesh N. Murthy and Mackenzie W. Mathis and Matthias Bethge},
    journal = {Nature Neuroscience},
    year = {2018},
    url = {https://www.nature.com/articles/s41593-018-0209-y}}

DeepLabCut2.x

 @article{NathMathisetal2019,
    title = {Using DeepLabCut for 3D markerless pose estimation across species and behaviors},
    author = {Nath*, Tanmay and Mathis*, Alexander and Chen, An Chi and Patel, Amir and Bethge, Matthias and Mathis, Mackenzie W},
    journal = {Nature Protocols},
    year = {2019},
    url = {https://doi.org/10.1038/s41596-019-0176-0}}

DeeperCut

@article{insafutdinov2016eccv,
    title = {DeeperCut: A Deeper, Stronger, and Faster Multi-Person Pose Estimation Model},
    author = {Eldar Insafutdinov and Leonid Pishchulin and Bjoern Andres and Mykhaylo Andriluka and Bernt Schiele},
    booktitle = {ECCV'16},
    url = {http://arxiv.org/abs/1605.03170}}

Google Active Learning Playground

@misc{activelearningplayground,
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/google/active-learning}}}