whacc 1.4.3

Automatic and customizable pipeline for creating a CNN + light GBM model to predict whiskers contacting objects

WhACC is a tool for automated touched image classification.

Many neuroscience labs (e.g. Hires Lab) use tasks that involve whisker active touch against thin movable poles to study diverse questions of sensory and motor coding. Since neurons operate at temporal resolutions of milliseconds, determining precise whisker contact periods is essential. Yet, accurately classifying the precise moment of touch is time-consuming and labor intensive.

Walkthrough: Google CoLab

Single example trial lasting 4 seconds. Example video (left) along with whisker traces, decomposed components, and spikes recorded from L5 (right). How do we identify the precise millisecond frame when touch occurs?

Original 2048 output features extracted from the penultimate layer of the initial ResNet50 V2 model, clustered for emphasize

Flow diagram of WhACC video pre-processing and design implementation

Touch frame scoring and variation in human curation

Data selection and model performance

Feature engineering and selection

WhACC shows expert human level performance

WhACC can be retrained on a small subset to account for data drift over time or different datasets (see GUI below)

WhACC GUI: used to curate automatically selected subset of data for optimal performance

Use left and right arrows to move through images, use up to label as touch (green) and down to label as not-touch (red)

Code contributors:

WhACC code and software was originally developed by Phillip Maire and Jonathan Cheung in the laboratory of Samuel Andrew Hires.