Skip to main content

Deep Learning Optimizer Benchmark Suite

Project description

# DeepOBS - A Deep Learning Optimizer Benchmark Suite

![DeepOBS](docs/deepobs_banner.png “DeepOBS”)

[![PyPI version](https://badge.fury.io/py/deepobs.svg)](https://badge.fury.io/py/deepobs) [![Documentation Status](https://readthedocs.org/projects/deepobs/badge/?version=stable)](https://deepobs.readthedocs.io/en/latest/?badge=stable) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

DeepOBS is a benchmarking suite that drastically simplifies, automates and improves the evaluation of deep learning optimizers.

It can evaluate the performance of new optimizers on a variety of real-world test problems and automatically compare them with realistic baselines.

DeepOBS automates several steps when benchmarking deep learning optimizers:

  • Downloading and preparing data sets.

  • Setting up test problems consisting of contemporary data sets and realistic deep learning architectures.

  • Running the optimizers on multiple test problems and logging relevant metrics.

  • Reporting and visualization the results of the optimizer benchmark.

![DeepOBS Output](docs/deepobs.jpg “DeepOBS_output”)

The code for the current implementation working with TensorFlow can be found on [Github](https://github.com/fsschneider/DeepOBS). A PyTorch version is currently developed (see News section below).

The full documentation is available on readthedocs: https://deepobs.readthedocs.io/

The paper describing DeepOBS has been accepted for ICLR 2019 and can be found here: https://openreview.net/forum?id=rJg6ssC5Y7

If you find any bugs in DeepOBS, or find it hard to use, please let us know. We are always interested in feedback and ways to improve DeepOBS.

## News

We are currently working on a new and improved version of DeepOBS, version 1.2.0. It will support PyTorch in addition to TensorFlow, has an easier interface, and many bugs ironed out. You can find the latest version of it in [this branch](https://github.com/fsschneider/DeepOBS/tree/v1.2.0-beta0).

A pre-release, version 1.2.0-beta0, will be available shortly and a full release is expected in a few weeks.

Many thanks to [Aaron Bahde](https://github.com/abahde) for spearheading the developement of DeepOBS 1.2.0.

## Installation

pip install deepobs

We tested the package with Python 3.6 and TensorFlow version 1.12. Other versions of Python and TensorFlow (>= 1.4.0) might work, and we plan to expand compatibility in the future.

If you want to create a local and modifiable version of DeepOBS, you can do this directly from this repo via

pip install -e git+https://github.com/fsschneider/DeepOBS.git#egg=DeepOBS

for the latest stable version, or

pip install -e git+https://github.com/fsschneider/DeepOBS.git@v1.2.0-beta0#egg=DeepOBS

to get the preview of DeepOBS 1.2.0.

Further tutorials and a suggested protocol for benchmarking deep learning optimizers can be found on https://deepobs.readthedocs.io/

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

deepobs-1.1.2.tar.gz (55.9 kB view details)

Uploaded Source

Built Distribution

deepobs-1.1.2-py3-none-any.whl (206.7 kB view details)

Uploaded Python 3

File details

Details for the file deepobs-1.1.2.tar.gz.

File metadata

  • Download URL: deepobs-1.1.2.tar.gz
  • Upload date:
  • Size: 55.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.3

File hashes

Hashes for deepobs-1.1.2.tar.gz
Algorithm Hash digest
SHA256 2a81cc03042e591b80aaf0fbba176dda53aff8c37534423a9a4306400fa5bb8a
MD5 44817feff74d17b7af5045ee5da876dc
BLAKE2b-256 37bc691f400f15fe12feedcd9046c0516f84c81b281fa4ed57fe43592bc2172a

See more details on using hashes here.

File details

Details for the file deepobs-1.1.2-py3-none-any.whl.

File metadata

  • Download URL: deepobs-1.1.2-py3-none-any.whl
  • Upload date:
  • Size: 206.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.3

File hashes

Hashes for deepobs-1.1.2-py3-none-any.whl
Algorithm Hash digest
SHA256 d2662ef1712033b996f32c340076e0f604e9a2d50530f02f7f6894504be27e9f
MD5 826013ebaccb88076a537f67dc9d31e9
BLAKE2b-256 7af80ee63f1095eba0f3062d8720fcaa41ef4689e7a8ce66be81177741a2898c

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page