Skip to main content

AI Benchmark is an open source python library for evaluating AI performance of various hardware platforms, including CPUs, GPUs and TPUs.

Project description

AI Benchmark Alpha is an open source python library for evaluating AI performance of various hardware platforms, including CPUs, GPUs and TPUs. The benchmark is relying on TensorFlow machine learning library, and is providing a lightweight and accurate solution for assessing inference and training speed for key Deep Learning models.

In total, AI Benchmark consists of 42 tests and 19 sections provided below:

  1. MobileNet-V2  [classification]
  2. Inception-V3  [classification]
  3. Inception-V4  [classification]
  4. Inception-ResNet-V2  [classification]
  5. ResNet-V2-50  [classification]
  6. ResNet-V2-152  [classification]
  7. VGG-16  [classification]
  8. SRCNN 9-5-5  [image-to-image mapping]
  9. VGG-19  [image-to-image mapping]
  10. ResNet-SRGAN  [image-to-image mapping]
  11. ResNet-DPED  [image-to-image mapping]
  12. U-Net  [image-to-image mapping]
  13. Nvidia-SPADE  [image-to-image mapping]
  14. ICNet  [image segmentation]
  15. PSPNet  [image segmentation]
  16. DeepLab  [image segmentation]
  17. Pixel-RNN  [inpainting]
  18. LSTM  [sentence sentiment analysis]
  19. GNMT  [text translation]

For more information and results, please visit the project website: http://ai-benchmark.com/alpha

Installation Instructions

The benchmark requires TensorFlow machine learning library to be present in your system.

On systems that do not have Nvidia GPUs, run the following commands to install AI Benchmark:

pip install tensorflow
pip install ai-benchmark

If you want to check the performance of Nvidia graphic cards, run the following commands:

pip install tensorflow-gpu
pip install ai-benchmark

Note 1: If Tensorflow is already installed in your system, you can skip the first command.

Note 2: For running the benchmark on Nvidia GPUs, NVIDIA CUDA and cuDNN libraries should be installed first. Please find detailed instructions here.

Getting Started

To run AI Benchmark, use the following code:

from ai_benchmark import AIBenchmark
benchmark = AIBenchmark()
results = benchmark.run()

Alternatively, on Linux systems you can type ai-benchmark in the command line to start the tests.

To run inference or training only, use benchmark.run_inference() or benchmark.run_training().

Advanced settings

AIBenchmark(use_CPU=None, verbose_level=1):

use_CPU={True, False, None}:   whether to run the tests on CPUs  (if tensorflow-gpu is installed)

verbose_level={0, 1, 2, 3}:   run tests silently | with short summary | with information about each run | with TF logs

benchmark.run(precision="normal"):

precision={"normal", "high"}:   if high is selected, the benchmark will execute 10 times more runs for each test.


Additional Notes and Requirements

GPU with at least 2GB of RAM is required for running inference tests / 4GB of RAM for training tests.

The benchmark is compatible with both TensorFlow 1.x and 2.x versions.

Contacts

Please contact andrey@vision.ee.ethz.ch for any feedback or information.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for ai-benchmark, version 0.1.1
Filename, size File type Python version Upload date Hashes
Filename, size ai_benchmark-0.1.1-py2-none-any.whl (21.5 MB) File type Wheel Python version py2 Upload date Hashes View hashes
Filename, size ai_benchmark-0.1.1-py3-none-any.whl (21.5 MB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size ai-benchmark-0.1.1.tar.gz (21.4 MB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page