norman-ai 0.1.0

A package for automation of the novel object recognition test.

The NORMAN System

The Novel Object Recognition Mouse Analysis Network (NORMAN) system is a package for automation of the novel object recognition behavioural assay in neuroscience studies.

PyPi link: https://pypi.org/project/norman-ai/

This is part of work that will be presented as a poster at the BNA Festival 2021 which runs from the 12-15th of April. It was developed as part of an MSci degree at the University of Dundee. The package and repository will be updated with video outlines and other details shortly, in the run up to the BNA festival.

Author: Oluwaseyi Jesusanmi
Twitter: @neuroseyience

---

Contents

1. System Overview
2. System Requirements
3. Installation guide
4. Video to DI walkthrough
5. Function details
6. Using Deeplabcut with NORMAN
7. Glossary

System Overview

The novel object recognition test (NORT) compares the time a mouse spends with a new/novel object and a familiar object in order to assess a mouses memory. The quantitative output of the test is reffered to as the discrimination index (DI), and is what many behavioural neuroscience studies use. It is calculated using (tN - tF)/(tN + tF), where tN is time spent with novel object and tF is time spent with familiar object.

The NORMAN system can automatically extract the DI and other metrics from a video of a NORT. The general steps in functioning of the system are as follows:

1. Use a deeplabcut network to track a mouse's position across a NORT video.
2. Automatically find object locations/spatial information.
3. Calculate relative positions of mouse to each object for every video frame.
4. Use an artificial neural network that I built and trained to understand mouse-object interaction (NORMAN) to label each frame with the relevant interaction.
5. Extract metrics, and visualise NORMAN's labelling.

The following guide uses Anaconda and conda for environment mangement, with python for programming. The program used for environment management was anaconda prompt, for quick python access I used ipython within anaconda prompt, and for in depth script-writing and development I used spyder 4. Other programs are available.

The guide assumes a very basic familiarity with anaconda/conda and python, but a user interface is available for those not comfortable coding. If help is needed feel free to contact me, I'd be happy discuss, make changes, and/or post more tutorials. I have also included a glossary in this document for some terms used.

System requirements and recommendations

Operating System - Windows 10 or 11. Note: All the packages used in this project have versions avaialable for Linux and Mac, so versions supporting these operating systems could be considered with sufficient interest.

GPU - For sufficient processing speed you must have a GPU. You would require an Nvidia GPU that is CUDA compatible and has at least 8Gbs of VRAM to work reliably well on both the Deeplabcut and NORMAN parts of the process.

RAM - Recommend 16Gbs

CPU - Any modern CPU multicore CP.

Tested on Windows 10 laptop i7-9750h, 16Gbs of Ram, Nvidia RTX 2060 mobile with 6Gbs of VRAM. On this hardware, I tested 5 minute novel object recognition video at 30fps and 312x480 resolution. The deeplabcut tracking step took ~2 minutes 15 seconds, and the NORMAN system analysis step took ~10 seconds. Also tested with Windows 11 laptop Ryzen 9 6900HX, 16Gbs Ram, Nvidia RTX 3070ti mobile with 8Gbs Ram.

Note on previous compatibility: The NORMAN system previously used Tensorflow-directml machine learning API. Tensorflow-directml has simple installation, and allows tensorflow use with hardware accelleration on ANY Directx12 capable GPU on a computer runnning Windows. That includes AMD, Nvidia, and even Intel GPUs. However it was still in active development, and at the time of writing it does not have a Tensorflow 2.X version currently available. So it is not compatible with the current versions of Deeplabcut that I reccommend to use with the NORMAN system.

Installation guide

This guide assumes working knowledge of the conda/anaconda environment management system. By the end of this section you should have an environment that works with deeplabcut, NORMAN, with GPU hardware accelleration through tensorflow-GPU utilising Nvidia CUDA. You can manage conda environments using the click-through user interface in Anaconda Navigator, or you can use the command line in anaconda prompt. This guide I will be using the anaconda prompt command line interface. For conda environment management commands, refer to this conda cheat sheet.

Firstly install Nvidia CUDA 11.4 on your system from the official website https://developer.nvidia.com/cuda-toolkit-archive. Many other versions may work but 11.4 has beeen tested. After this open anaconda prompt and enter the following commands.

#Create initial environment with python and tensorflow installed
# norman_dlc_cuda can be any name you choose
conda create -n norman_dlc_cuda python=3.8 tensorflow-gpu=2.5
# Activate the environment so you can install more packages on it
conda activate norman_dlc_cuda
# install the animal tracking library
pip install "deeplabcut[gui]"==2.3.0
#install norman
pip install norman-ai
#install required version of numpy
conda install numpy==1.23.4

Now that installation is complete, perform tests to see if installation was successful using ipython. Ipython is a python program useful for working at the command line.

#open the ipython program
ipython

#check if your graphics card is recognised by tensorflow after installation

import tensorflow as tf
print("Num GPUs Available: ", len(tf.config.list_physical_devices("GPU")))

#check if norman was installed
import norman_ai.norman_functions as nf
nf.run_gui()
#a user interface should appear

Video to DI walkthrough

This section will take you through the basic steps required to get a DI from a video and to visualise NORMAN'S labelling, both using the provided user interface and with code. Make sure you are in the conda environment with norman and deeplabcut installed before proceeding. Please note that the user interface is primarily for demonstration purposes. To fully take advantage of NORMAN and process in batches for your own application, the code method is what you should use.

User interface method

This uses the video file, a premade deeplabcut tracking file and an option norman model file in the demo folder of the project repository.

#open the ipython program
ipython

#import norman
import norman_ai.norman_functions as nf
#open graphical user interface (gui)
nf.run_gui()

1. Input the video file "test_79.mp4" by clicking on "please select a video", a file selection window will open.
2. Select the corresponding deeplabcut tracking csv "test_79_poses_filtered.csv" file by clicking on "please select a pose file".
3. Enter which object is the video is novel to the mouse.
4. Click accept and run video analysis. (Selecting the norman model is optional).
5. Results will be in the bottom left.
6. To show whether the find_objects function worked correctly, press the "Display object detection" button.
7. Press the "Visualisation of NORMAN labelling" button to produce a video showing how NORMAN labelled the video. The video will be in the same directory as the original video, with "norman" added to the name.

• Play the video and see how NORMAN does.

Code-based method

To follow the code walkthrough set the working directory to the demo folder in the project repository.

#import norman functions
import norman_ai.norman_functions as nf
#import deeplabcut functions
import deeplabcut as dlc

#get video name
video = "test_79.mp4"

Deeplabcut needs a path to a deeplabcut project config file in order for its functions to work. A pretrained network is provided in the demo folder. Please note if using a full path to the config file, it may be different to the paths in the tutorial depending on where you store the repository.

#get the deeplabcut project config file location.
#the r makes the string be treated as a raw string
path_cfg = r"\nort_demo-Jesusanmi-2019-09-26\config.yaml"

#analyse the video and save the results to a csv
dlc.analyze_videos(path_cfg, video, videotype="mp4", save_as_csv=(True))

#filter the predcitions from video analysis for smoother tracking
dlc.filterpredictions(path_cfg, video, save_as_csv=(True))

From here NORMAN functions use information from the video file combined with the csv file for analysis.

#get path to pose document
poses = r"test_79DLC_resnet_50_nort_demoSep26shuffle1_50000_filtered.csv"

#create norkid python object
# stating the novel object is the left object in the experiment, if the right object is novel state "right"
x = nf.norkid(video, poses, "left")

#show the discrimination index, the time spent with left object, time spent with right object
print("DI:"+str(x.di)+", Time left:"+str(x.tl)+ ", Time right:"+ str(x.tr))

#make an video visualisation of norman labelling the mouse video
#this is optional as not every video result will need visualisation
x.draw_vid()

If you have trouble with this demo, please see the function details, which contain code examples and guidance on other functions in the NORMAN package.

When using NORMAN with your own data, the most common problem is related to the find_objects() function, as not all NORT mouse chambers are compatible with the object detection algorithm. If this is the case you can use a conditional argument to draw the location of the experimental objects, and all the other parts of analysis will still be done automatically.

Hold the "r" key to reset the drawing and hold the escape key to exit once the drawing is complete.

#if you wish to mark object locations yourself use draw_objects as true
#hold r to reset the drawing
#press escape to exit the window once the drawing is complete
x = nf.norkid(video, poses, "left", draw_obj=True)

#show the discrimination index, the time spent with left object, time spent with right object
print("DI:"+str(x.di)+", Time left:"+str(x.tl)+ ", Time right:"+ str(x.tr))

#make an video visualisation of norman labelling the mouse video
#this is optional as not every video result will need visualisation
x.draw_vid()

Function details

Though very few NORMAN functions are needed to extract a discrimination index from a video, the NORMAN system has more functions for working with NORT images and videos. In this section I will explain how to use various functions in the library, with their use-cases and code examples. Where relevant I will also include information on how/why I made the function for context. Code examples below were done in the “demo” folder of the NORMAN project repository, using the “test_79.mp4” video as the source data. The top of each code example with show the arguments of the function and their defaults when applicable. For the full code of each function see the norman_functions.py file, and for the help doc strings use the python default function help().

#norman help example
import norman_ai.norman_functions as nf
# ask for documentation for the function
help(nf.find_objects)

Norkid python class, object parameters and methods

The "norkid" class handles all the processes to analyse a video, while holding the data for a video in one place. To create a norkid object you need a path to a video, a tracking csv, and which experiment object is the novel object (left or right). A norkid object stores: an image with the mouse filtered out, the find object output, the relative position function output, the labels produced by the NORMAN model, the DI and related metrics. These properties can be used in conjunction with other functions and libraries, while easing the debugging process.

#%% norkid object example
import norman_ai.norman_functions as nf
from matplotlib import pyplot as plt

#get video name, norman model file and deeplabcut tracking file
video = "test_79.mp4"
model_path = "norman_model1.h5"
pose_file = "test_79_poses_filtered.csv"

#make a norkid object
x = nf.norkid(video, pose_file, model_path, "left")

# view name of input video
x.video_name
#show the median filtered image
plt.imshow(x.median_img)

# show find objects outline image
plt.imshow(x.fo_img)

#view object location datafrane
x.object_locs

#view relative position dataframe
x.relative_pos

#view norman labels per frame
x.labels

#show the DI, seconds spent with left and right objects, and the video fps
print(round(x.di, 2), round(x.tl , 2), round(x.tr,2) , round(x.fps, 2))
-0.36 20.75 43.83 18.07

# make a norman labelled video with a default name (video name + _norman)
x.draw_vid()
# make a norman labelled video with a specified name
x.draw_vid(out_name="my_norman_vid.mp4")

Frame extractor – ext_frame()

Extract a frame or multiple frames from a video. Outputs an image from a video as an image array, a file, or a folder image files. Often when working with video data, specific frames may need to be checked or analysed, as many functions work on a frame-by-frame basis.

# Function arguments = ext_frame(vid_name, frame_index, out_name=False)
# Extract frame example 
import norman_ai.norman_functions as nf
# set video name
my_video = “test_79.mp4”

# Extract a single frame to produce an image array
image_array = nf.ext_frame(my_video, 42)

# Extract a frame and save as an image file
nf.ext_frame(my_video , 42, out_name = "my_frame.png")

# Extract frames 42 to 150 to the current working directory
nf.ext_frame(my_video,  (42, 150))

# Extract frames 42 to 150 to a new folder
nf.ext_frame(my_video, (42, 150), out_name = "many_frames")

Resolution changer – reso_change()

Changes the resolution of a video to a specified value on the x and y axis. Creates a new video, does not affect the data of the original video.

# Function Arguments: reso_change(input_vid, out_vid, res=(640, 480))
# Resolution change example.
#Please note new video is created, original video is not affected
import norman_ai.norman_functions as nf
# set video name
my_video = “test_79.mp4”

#change resolution of video to default value of 640x480
nf.reso_change(my_video, "new_res.mp4")

#change resolution of video to a new arbitrary value
nf.reso_change(my_video, "new_res.mp4", res=(789,456))

Moving object remover/filter – median_filt_video()

This function removes the mouse from the video and produces an image of the bare experiment box and objects. This must be done before using the find_objects() function for object detection, as the mouse will interfere with the object finder. It will remove any moving object from a video given enough frames.

#Function arguments: median_filt_video(video_name, show=False, out_file=False, select_no=15)
#%% Median filter example
import norman_ai.norman_functions as nf
from matplotlib import pyplot as plt
# set video name
my_video = “test_79.mp4”

#show what a single frame looks like before filtering
plt.imshow(nf.ext_frame(my_video, 42))

#use median filter with default 15 frames, show the result, output to variable and to file
x = nf.median_filt_video(my_video, show=True, out_file="filtered.png")

#use median filter with specified number of frames, output to variable
x = nf.median_filt_video(my_video, select_no=20)

Object localiser – find_objects()

This automatically finds the 2 experiment objects and outputs the locations, area and outline of each of them. The values are used later to calculate proximity of the mouse to objects. This function should be used after the median filter has been used to create a mouse-free image. So far in testing it reliably finds the location of objects on videos where the floor of the test chamber is plain. An alternative version of the function is under exploration for boxes with patterned floors or large amounts of debris, as this affects the edge detection algorithms.

# Function Arguments: find_objects(image_or_path, show=False, img_out=False, im_write=False)
# Find objects example
import norman_ai.norman_functions as nf
from matplotlib import pyplot as plt
# set video name
my_video = “test_79.mp4”

#use median filter to generate a mouse-free image array or file 
box_image = nf.median_filt_video(my_video)
nf.median_filt_video(my_video, out_file="box_img.png")

#show image using matplotlib. Note it may show in false colour
plt.imshow(box_image)

#use the object detector on an image array, show the result, save the output as pandas dataframe
x = nf.find_objects(box_image, show=True)

#use object detector on an image file, show the result, save output as pandas dataframe
x = nf.find_objects("box_img.png", show=True)

#use object detector and output resulting image array with dataframe
x, image = nf.find_objects(box_image, img_out=True)

#use object detector and write the output image as a file
x = nf.find_objects(box_image, im_write="outlines.png")

Video maker from multiple images – make_video()

This function takes in a directory of images, and outputs a video created from these images at any specified framerate.

#Function arguments: make_video(image_folder, new_vid, fps=25)
# Make video example
import norman_ai.norman_functions as nf

# make a new video from a folder of images with an fps of 25
nf.make_video("many_frames", "vid_new.mp4")

# make a new video from a folder of images with any fps
nf.make_video("many_frames", "vid_new.mp4", fps=60)

Making predictions from position data – label_vid()

Prediction in the case of the NORMAN system refers to the NORMAN network model taking relative positional information per-frame, then determining what the mouse is paying attention to in each frame. In the output array, neither=0, left object =1, right object=2. These labels are used for calculating the DI or other useful metrics. Since the labels are per-frame, many NORT attention metrics can be calculated, including latency to first interaction and change in interaction amount over the time course of an experiment.

#Function arguments: label_vid(model_path, relative_pos)
# Label vid example
import norman_ai.norman_functions as nf

#input video name
video = "test_79.mp4"
#input path to pose file 
pose_file = "test_79_poses_filtered.csv"

#extract the median image 
median_img = nf.median_filt_video(video)
# extract object locations 
object_locs = nf.find_objects(median_img)
# calculate relative position of mouse to objects
relative_pos = nf.rel_pos(pose_file, object_locs)
#get labels produced by the prediction by norman
labels = nf.label_vid(relative_pos)  

Making a NORMAN labelled video – draw_vid()

The draw_vid() function visualises the NORMAN model predictions (see figure). This is useful for assessing whether the system is working correctly as it can be judged by eye. It is also useful when presenting data to show how the NORMAN system works.

# Function arguments: draw_vid(y_labels, input_vid, out_name=False)
# Draw vid example
import norman_ai.norman_functions as nf

#input video name
video = "test_79.mp4"
#input path to norman model
model_path = "norman_model1.h5"
#input path to pose file 
pose_file = "test_79_poses_filtered.csv"

#extract the median image 
median_img = nf.median_filt_video(video)
# extract object locations 
object_locs = nf.find_objects(median_img)
# calculate relative position of mouse to objects
relative_pos = nf.rel_pos(pose_file, object_locs)
#get labels produced by the prediction by norman
labels = nf.label_vid(model_path, relative_pos)  

#make norman labelled video, with default naming of the original video name +_norman saved to the current working directory
nf.draw_vid(labels, video)

#make norman labelled video, with specified name saved to specified directory.
nf.draw_vid(labels, video, out_name="labelled_vid.mp4")

Calculating discrimination index – calc_di()

This function only produced the DI, time spent with each object and the fps.

# calc di example
import norman_ai.norman_functions as nf

#input video name
video = "test_79.mp4"

#extract labels from video
pose_file = "test_79_poses_filtered.csv"
median_img = nf.median_filt_video(video) 
object_locs = nf.find_objects(median_img)
relative_pos = nf.rel_pos(pose_file, object_locs)
labels = nf.label_vid(relative_pos) 

# calculate and store just the di of a video.
# we are saying the novel object is the left one
di = nf.calc_di(video, labels, "left", ret_all = False)
# store the di, time with left, time with right and fps (default)
#we are saying the novel object is the right one
di, tl, tr, fps = nf.calc_di(video, labels, "right")

#show results rounded to 2 decimal places
print(round(di, 2), round(tl , 2), round(tr,2) , round(fps, 2))
0.35 20.81 43.66 18.07

Converting maze project csv into one compatible with NORMAN

Note: This function is only applicable to one specific project. This is a function to convert Deeplabcut output that was recorded from the bottom of the mouse to one compatible with the rest of the NORMAN system. It assumes the nose, mid and tailbase of the mouse marked were marked in Deeplabcut, and uses these to infer where the ears are, then creates a new CSV based on this. This csv file can then be used as a normal pose file in the system going forwards. It would be advisable to use this function in a loop to do a lot of files at once.

#get name of the file you want to convert
maze_pose_file = r"maze_dlc_output.csv"
#name of the the new file you want to create
new_file_name = "converted_file.csv"
#convert the file
nf.maze_file_convert(maze_pose_file, new_file_name)

Using Deeplabcut with NORMAN.

Deeplabcut is a python library for marker-less tracking of animals using artificial neural networks. It provides a variety of functions centred around streamlining the process of tracking animals. The general workflow of the library is to use a neural network that has been pretrained on animal movement, train it to recognise features on the animal you wish to track by labelling images, then use the trained network to track animal movement on a selection of videos. For in depth details please refer to the official deeplabcut tutorials on github. Here I will briefly discuss points that are relevant for using Deeplabcut in tandem with the NORMAN system.

Installation – The tutorials recommend tensorflow-gpu installation for using Deeplabcut with hardware accelleration. But the previously discussed tensorflow-directml based environment allows a lot easier installation when using a GPU and allows you to have much more customizable environments, while remaining compatible with deeplabcut.

Config file – When you start a new deeplabcut project it creates a folder with a “config.yaml” file. This config file is what all the functions use to find other files in the directory, such as the location of the trained models. The entire project folder can be moved, copied and shared. As long as the information in the config file is up do date, all the functions should work as normal.

Training a deeplabcut model – When labelling frames to train Deeplabcut to track mice or rats, please ensure the labels are added in the right order with the correct text. This is to ensure the output csv from deeplabcut analysis is compatible with the NORMAN system. The list of labels is: nose, l_ear, r_ear, tailbase. L_ear refers to left ear, r_ear refers the right ear, and tailbase refers to where the tail connects to the main body of the mouse. If you are training a Deeplabcut network with a GPU that has less than 8Gbs of video memory (VRAM), you may run into memory allocation errors. To solve this you must change the session parameters to allow a procedural change in memory allocation during training . The following code implements this:

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)

The Deeplabcut model zoo – Deeplabcut has a growing “model zoo”, where you can download pre-trained models to use for animal tracking. These are very convenient as the most time consuming step (training and evaluating the model), is eliminated. There is no pretrained mouse tracking model available yet, but when one is released, this should be used and NORMAN will be given a function to accept the csv files produced when using this model.

Video resolution - The processing time/resolution trade off must also be considered. Higher resolution can mean higher accuracy as there is more detail visible, but higher resolutions can greatly increase the processing time for the deeplabcut tracking step. The point of diminishing returns from resolution differs depending on the application. For example if you are tracking the nose and ears of a mouse, a very low resolution of just 480p would provide good tracking accuracy. Whereas if you were tracking whiskers at that same resolution with the mouse equidistant from the camera, the whiskers will be indistinguishable and the accuracy would suffer tremendously.

Glossary

Array – A data type for storing sequences of a variety of dimensions, usually in the numpy library. It can be used to store a simple sequence of numbers, or something more complex like an image by representing each pixel colour with a number in a table-like structure. Array processing is incredibly fast, making them useful for matrices calculations.

Class – In python, a class refers to a type of object and is used as a layout on which to build an object. An object in python has properties (parameters and characteristics), and methods (types of functions) associated with it. For an imaginary example, say a pet golden retriever was a python object. It would have properties such as height, weight, breed, fur colour etc. It could have methods such as run, play fetch, eat, sleep etc. These methods could also affect some of the object properties, for example the “eat” method may increase the dog object’s weight. The class for this example would be a dog class. It would be a common set of properties and methods of which the exact values of which will be different for each new dog object that is made. Python is an object orientated programming language, meaning that many data and variable are often stored as objects/object parameters, with methods which can be applied to them. For a practical implementation of a class and object, see the norkid class explanation in section 4.

Computer vision – A field of computer science programming that aims to extract information from images and video. This often works by converting images into an array of numbers which represent pixels, and using algebraic functions on these numbers to extract the necessary information. Recently much of computer vision research utilises artificial neural network techniques as they can apply incredibly complex mathematical functions to images automatically, if given enough training data. Frame/video frame – Videos are made up of many images in sequence, each individual image is called a frame. Most standard video is shot at about 30 frames per second, meaning each second of video is made out of 30 images.

GPU/integrated GPU – A graphical processing unit is a specialised computer component made for processing highly parallel mathematical problems. Originally produced for computer graphics and video-games, it was discovered that the parallel nature of the hardware is well suited to running artificial neural network operations. GPUs are often parts of graphics cards, while integrated GPUs are less powerful devices that are integrated into CPU chipsets, usually for devices without dedicated graphics cards.

Library/python library – Sets of reusable code. A library often includes a series of functions that are helpful for a specific type of programming problem. For example pandas is a tabular data processing library, which includes sets of functions useful for processing tabular data.

Object – See entry for “Class”.

Package – Package is a collection of code similar to a library. It is referred to as a package when it is released in a distributable form that can be installed into an environment.

Version control – A method of archiving previous version of a coding project, commonly the git package is used to do this. Using this method all files and all changes to files are stored. This means that if the code breaks, you can roll-back to a previous version where the code was working. It is also important for scientific reproducibility, since all changes are tracked, any mistakes or instances of data being changed in a fraudulent manner can be tracked.

Virtual development environment – A partition on a computer where many libraries and packages can be installed, in a way that other aspects of the computer are not affected. This allows developers to work on multiple different projects that require different packages without running into as many compatibility errors.