Command-line utility for searching and filtering videos based on objects that appear in them using convolutional neural networks
## Thingscoop: Search and filter videos based on their content
Thingscoop is a command-line utility for analyzing videos semantically - that means searching, filtering, and describing videos based on objects, places, and other things that appear in them.
When you first run thingscoop on a video file, it uses a [convolutional neural network](https://en.wikipedia.org/wiki/Convolutional_neural_network) to create an "index" of what's contained in the every second of the input by repeatedly performing image classification on a frame-by-frame basis. Once an index for a video file has been created, you can search (i.e. get the start and end times of the regions in the video matching the query) and filter (i.e. create a [supercut](https://en.wikipedia.org/wiki/Supercut) of the matching regions) the input using arbitrary queries. Thingscoop uses a very basic query language that lets you to compose queries that test for the presence or absence of labels with the logical operators `!` (not), `||` (or) and `&&` (and). For example, to search a video the presence of the sky *and* the absence of the ocean: `thingscoop search 'sky && !ocean' <file>`.
Right now two convolutional neural network models are supported by thingscoop: `vgg_imagenet` (using the architecture described in ["Very Deep Convolutional Networks for Large-Scale Image Recognition"](http://arxiv.org/abs/1409.1556)) recognizes objects from the [ImageNet](http://www.image-net.org/) database, and `googlenet_places` (using the architecture described in ["Going Deeper with Convolutions"](http://arxiv.org/abs/1409.4842)) recognizes settings and places from the [MIT Places](http://places.csail.mit.edu/) database. You can specify which model you'd like to use by running `thingscoop models use <model>`, where `<model>` is either `vgg_imagenet` or `googlenet_places`. More models will be added soon.
Thingscoop is based on [Caffe](http://caffe.berkeleyvision.org/), an open-source deep learning framework.
1. Install ffmpeg: `brew install ffmpeg` (Mac OS X) or `apt-get install ffmpeg` (Ubuntu).
1. Follow the installation instructions on the [Caffe Installation page](http://caffe.berkeleyvision.org/installation.html).
2. Make sure you build the Python bindings by running `make pycaffe` (on Caffe's directory).
3. Set the environment variable CAFFE_ROOT to point to Caffe's directory: `export CAFFE_ROOT=[Caffe's directory]`.
4. Install thingscoop: `easy_install thingscoop` or `pip install thingscoop`.
#### `thingscoop search <query> <files...>`
Print the start and end times (in seconds) of the regions in `<files>` that match `<query>`. Creates an index for `<file>` using the current model if it does not exist.
$ thingscoop search violin waking_life.mp4
/Users/anastasis/Downloads/waking_life.mp4 148.000000 162.000000
/Users/anastasis/Downloads/waking_life.mp4 176.000000 179.000000
/Users/anastasis/Downloads/waking_life.mp4 180.000000 186.000000
/Users/anastasis/Downloads/waking_life.mp4 189.000000 190.000000
/Users/anastasis/Downloads/waking_life.mp4 192.000000 200.000000
/Users/anastasis/Downloads/waking_life.mp4 211.000000 212.000000
/Users/anastasis/Downloads/waking_life.mp4 222.000000 223.000000
/Users/anastasis/Downloads/waking_life.mp4 235.000000 243.000000
/Users/anastasis/Downloads/waking_life.mp4 247.000000 249.000000
/Users/anastasis/Downloads/waking_life.mp4 251.000000 253.000000
/Users/anastasis/Downloads/waking_life.mp4 254.000000 258.000000
####`thingscoop filter <query> <files...>`
Generate a video compilation of the regions in the `<files>` that match `<query>`. Creates an index for `<file>` using the current model if it does not exist.
<a href="https://www.youtube.com/watch?v=qe9GjrUJipY"><img width=600 src="resources/filter.png"></img></a>
#### `thingscoop preview <file>`
Create a window that plays the input video `<file>` while also displaying the labels the model recognizes on every frame.
<img width=800 src="resources/preview.jpg"></img>
#### `thingscoop describe <file>`
Print every label that appears in `<file>` along with the number of times it appears. Creates an index for `<file>` using the current model if it does not exist.
$ thingscoop describe koyaanisqatsi.mp4 -m googlenet_places
#### `thingscoop index <file>`
Create an index for `<file>` using the current model if it does not exist.
#### `thingscoop models list`
List all models currently available in Thingscoop.
$ thingscoop models list
googlenet_imagenet Model described in the paper "Going Deeper with Convolutions" trained on the ImageNet database
googlenet_places Model described in the paper "Going Deeper with Convolutions" trained on the MIT Places database
vgg_imagenet 16-layer model described in the paper "Return of the Devil in the Details: Delving Deep into Convolutional Nets" trained on the ImageNet database
#### `thingscoop models info <model>`
Print more detailed information about `<model>`.
$ thingscoop models info googlenet_places
Description: Model described in the paper "Going Deeper with Convolutions" trained on the MIT Places database
Dataset: MIT Places
#### `thingscoop models freeze`
List all models that have already been downloaded.
$ thingscoop models freeze
#### `thingscoop models current`
Print the model that is currently in use.
$ thingscoop models current
#### `thingscoop models use <model>`
Set the current model to `<model>`. Downloads that model locally if it hasn't been downloaded already.
#### `thingscoop models download <model>`
Download the model `<model>` locally.
#### `thingscoop models remove <model>`
Remove the model `<model>` locally.
#### `thingscoop models clear`
Remove all models stored locally.
#### `thingscoop labels list`
Print all the labels used by the current model.
$ thingscoop labels list
#### `thingscoop labels search <regexp>`
Print all the labels supported by the current model that match the regular expression `<regexp>`.
$ thingscoop labels search instrument$
bowed stringed instrument
### Full usage options
thingscoop search <query> <files>... [-o <output_path>] [-m <model>] [-s <sr>] [-c <mc>] [--recreate-index] [--gpu-mode]
thingscoop filter <query> <files>... [-o <output_path>] [-m <model>] [-s <sr>] [-c <mc>] [--recreate-index] [--gpu-mode] [--open]
thingscoop preview <file> [-m <model>] [--gpu-mode] [--min-confidence <ct>]
thingscoop describe <file> [-n <words>] [-m <model>] [--recreate-index] [--gpu-mode] [-c <mc>]
thingscoop index <file> [-m <model>] [-c <mc>] [-s <sr>] [-c <mc>] [--recreate-index] [--gpu-mode]
thingscoop labels list [-m <model>]
thingscoop labels search <regexp> [-m <model>]
thingscoop models list
thingscoop models freeze
thingscoop models current
thingscoop models use <model>
thingscoop models download <model>
thingscoop models remove <model>
thingscoop models info <model>
thingscoop models clear
--version Show version.
-h --help Show this screen.
-o --output <dst> Output file for supercut
-s --sample-rate <sr> How many frames to classify per second (default = 1)
-c --min-confidence <mc> Minimum prediction confidence required to consider a label (default depends on model)
-m --model <model> Model to use (use 'thingscoop models list' to see all models available)
-n --number-of-words <words> Number of words to describe the video with (default = 5)
--gpu-mode Enable GPU mode
--recreate-index Recreate object index for file if it already exists
--open Open filtered video after creating it (OS X only)
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size thingscoop-0.1.tar.gz (11.0 kB)||File type Source||Python version None||Upload date||Hashes View|