pymoten 0.1.2

Extract motion energy features from video using spatio-temporal Gabors

What is pymoten?

pymoten is a python package that provides a convenient way to extract motion energy features from video using a pyramid of spatio-temporal Gabor filters [1] [2]. The filters are created at multiple spatial and temporal frequencies, directions of motion, x-y positions, and sizes. Each filter quadrature-pair is convolved with the video and their activation energy is computed for each frame. These features provide a good basis to model brain responses to natural movies [3] [4].

Installation

Using pip, install the latest version from git:

pip install git+https://github.com/gallantlab/pymoten.git

Or the most recent release:

pip install pymoten

Getting started

Example using synthetic data

import moten
import numpy as np

# Generate synthetic data
nimages, vdim, hdim = (100, 90, 180)
noise_movie = np.random.randn(nimages, vdim, hdim)

# Create a pyramid of spatio-temporal gabor filters
pyramid = moten.get_default_pyramid(vhsize=(vdim, hdim), fps=24)

# Compute motion energy features
moten_features = pyramid.project_stimulus(noise_movie)

Simple example using a video file

import moten

# Stream and convert the RGB video into a sequence of luminance images
video_file = 'http://anwarnunez.github.io/downloads/avsnr150s24fps_tiny.mp4'
luminance_images = moten.io.video2luminance(video_file, nimages=100)

# Create a pyramid of spatio-temporal gabor filters
nimages, vdim, hdim = luminance_images.shape
pyramid = moten.get_default_pyramid(vhsize=(vdim, hdim), fps=24)

# Compute motion energy features
moten_features = pyramid.project_stimulus(luminance_images)

GPU acceleration

pymoten supports multiple computational backends. By default it runs on the CPU with NumPy, but it can also run on the GPU using PyTorch, which is much faster for large stimuli. The available backends are "numpy" (default), "torch", "torch_cuda" (NVIDIA GPUs), and "torch_mps" (Apple Silicon GPUs). The torch backends require PyTorch to be installed.

import moten
from moten.backend import set_backend

# Switch to a GPU backend (returns the backend module)
backend = set_backend("torch_cuda")

pyramid = moten.get_default_pyramid(vhsize=(vdim, hdim), fps=24)

# Move the stimulus onto the GPU, project, then bring the result back
stimulus_gpu = backend.asarray(luminance_images)
moten_features = pyramid.project_stimulus_batched(stimulus_gpu)
moten_features = backend.to_numpy(moten_features)

See the examples gallery for a complete walkthrough.

Cite as

Nunez-Elizalde AO, Deniz F, Dupré la Tour T, Visconti di Oleggio Castello M, and Gallant JL (2021). pymoten: scientific python package for computing motion energy features from video. Zenodo. https://doi.org/10.5281/zenodo.6349625

References

[1]

Adelson, E. H., & Bergen, J. R. (1985). Spatiotemporal energy models for the perception of motion. Journal of the Optical Society of America A, 2(2), 284-299.

[2]

Watson, A. B., & Ahumada, A. J. (1985). Model of human visual-motion sensing. Journal of the Optical Society of America A, 2(2), 322–342.

[3]

Nishimoto, S., & Gallant, J. L. (2011). A three-dimensional spatiotemporal receptive field model explains responses of area MT neurons to naturalistic movies. Journal of Neuroscience, 31(41), 14551-14564.

[4]

Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., & Gallant, J. L. (2011). Reconstructing visual experiences from brain activity evoked by natural movies. Current Biology, 21(19), 1641-1646.

---

A MATLAB implementation can be found here.