pyspacer 0.4.1

Spatial image analysis with pytorch and caffe backends.

PySpacer

This repository provide utilities to extract features from random point locations in images and then training classifiers over those features. It is used in the vision backend of https://github.com/coralnet/coralnet.

Spacer currently supports python >=3.8.

Overview

Spacer executes tasks as defined in messages. The messages types are defined in messages.py and the tasks in tasks.py. We also define several data-types in data_classes.py which define input and output types.

Refer to the unit-test in test_tasks.py for examples on how to create tasks.

Tasks can be executed directly by calling the methods in tasks.py. However, spacer also supports an interface with AWS Batch handled by env_job() in mailman.py.

Spacer supports four storage types: s3, filesystem, memory and url. Refer to storage.py for details. The Memory storage is mostly used for testing, and the url storage is read only.

Also take a look at config.py for settings and configuration.

Installation

The spacer repo can be installed in three ways.

• Using Docker -- the only option that supports Caffe.
• Local clone -- ideal for testing and development.
• Using pip install -- for integration in other code-bases.

Config

Spacer's config variables can be set in any of the following ways:

1. As environment variables; recommended if you pip install the package. Each variable name must be prefixed with SPACER_:
   • export SPACER_AWS_ACCESS_KEY_ID='YOUR_AWS_KEY_ID'
   • export SPACER_AWS_SECRET_ACCESS_KEY='YOUR_AWS_SECRET_KEY'
   • export SPACER_LOCAL_MODEL_PATH='/path/to/your/local/models'
2. As a Django setting; recommended for a Django project that uses spacer. Example code in a Django settings module:

   SPACER = {
       'AWS_ACCESS_KEY_ID': 'YOUR_AWS_KEY_ID',
       'AWS_SECRET_ACCESS_KEY': 'YOUR_AWS_SECRET_KEY',
       'LOCAL_MODEL_PATH': '/path/to/your/local/models',
   }
   

3. In a secrets.json file in the same directory as this README; recommended for Docker builds and local clones. Example secrets.json contents:

   {
     "AWS_ACCESS_KEY_ID": "YOUR_AWS_KEY_ID",
     "AWS_SECRET_ACCESS_KEY": "YOUR_AWS_SECRET_KEY",
     "LOCAL_MODEL_PATH": "/path/to/your/local/models"
   }
   

LOCAL_MODEL_PATH is required. The two AWS access variables are required unless spacer is running on an AWS instance which has been set up with aws configure. The rest of the config variables are optional; see CONFIGURABLE_VARS in config.py for a full list.

To debug your configuration, try opening a Python shell and run from spacer import config, then config.check().

Docker build

The docker build is the preferred build and the one used in deployment.

• Install docker on your system
• Create secrets.json as detailed above.
• Create folder /path/to/your/local/models for caching model files.
• Build image: docker build -t spacer:test .
• Run: docker run -v /path/to/your/local/models:/workspace/models -v ${PWD}:/workspace/spacer/ -it spacer:test

The -v /path/to/your/local/models:/workspace/models part will make sure the downloaded models are cached to your host storage. which makes rerunning stuff much faster.

The -v ${PWD}:/workspace/spacer/ mounts your current folder including secrets.json so that the container has the right permissions.

The last step will run the default CMD command specified in the dockerfile (unit-test with coverage). If you want to enter the docker container run the same command but append bash in the end.

Pip install

• pip install pyspacer
• Set environmental variables.

Local clone

• Clone this repo
• pip install -r requirements.txt

If using Windows: turn Git's autocrlf setting off before your initial checkout. Otherwise, pickled classifiers in spacer/tests/fixtures will get checked out with \r\n newlines, and the pickle module will fail to load them, leading to test failures. However, autocrlf should be left on when adding any new non-pickle files.

Code coverage

If you are using the docker build or local install, you can check code coverage like so:

    coverage run --source=spacer --omit=spacer/tests/* -m unittest    
    coverage report -m
    coverage html