OSML Model Runner Test

This package contains the integration tests for OSML application

Table of Contents

• Getting Started
  • Prerequisites
  • Installation Guide
  • Documentation
  • Build and Local Testing
  • Running LoadTest
• Support & Feedback
• Security
• License

Getting Started

Prerequisites

First, ensure you have installed the following tools locally

1. aws cli
2. tox

Installation Guide

1. Clone osml-model-runner-test package into your desktop

git clone https://github.com/aws-solutions-library-samples/osml-model-runner-test.git

1. Run tox to create a virtual environment

cd osml-model-runner-test
tox

Documentation

You can find documentation for this library in the ./doc directory. Sphinx is used to construct a searchable HTML version of the API documents.

tox -e docs

Local Testing

Credentials

Credentials from the user's account are volume mounted into the container's root directory.

Processing an image:

You can run the integration tests against your dev account by exporting the required parameters and using the pytest CLI by using the python script bin/process_image.py. Remember to load up your AWS credentials into your terminal, please follow this guide on how to load your aws credentials.

python bin/process_image.py --image <image type> --model <model type>

Examples:

python3 bin/process_image.py --image small --model centerpoint

python3 bin/process_image.py --image meta --model centerpoint

python3 bin/process_image.py --image large --model flood

python3 bin/process_image.py --image tile_tif --model aircraft

To print out the usage for python script, execute:

python3 bin/process_image.py --help

To execute the integration test, exclude --skip_integ from the command line interface. It is essential that the images and models listed in the table below are aligned accurately for the test to succeed. Conversely, by adding --skip_integ to the CLI, all comparison checks will be bypassed, rendering the table irrelevant for testing purposes.

Running LoadTest

You can run the load test against your dev account and be able to determine the cost and the performance. Please advise it can potentially rack up your AWS bills!

Examples:

python3 bin/run_load_test.py --periodic_sleep 60 --processing_window 1

To print out the usage for this load test script, execute:

python3 bin/run_load_test.py --help

Support & Feedback

To post feedback, submit feature ideas, or report bugs, please use the Issues section of this GitHub repo.

If you are interested in contributing to OversightML Model Runner, see the CONTRIBUTING guide.

Security

See CONTRIBUTING for more information.

License

MIT No Attribution Licensed. See LICENSE.