mlbench-core: Distributed Machine Learning Benchmark Core Library

[](https://travis-ci.com/mlbench/mlbench-core) [](https://mlbench.readthedocs.io/projects/mlbench_core/en/latest/?badge=latest) [](https://codecov.io/gh/mlbench/mlbench-core)

MLBench is a Benchmarking Framework for Distributed Machine Learning algorithms.

This repository contains the core Python library for MLBench which is used to share code between Benchmark implementations as well as for communication with the dashboard.

For more information refer to the [MLBench Core Documentation](https://mlbench.readthedocs.io/projects/mlbench_core/en/stable/api.html) or the [Main Documentation](https://mlbench.readthedocs.io/)

# Changelog

## [v2.4.0](https://github.com/mlbench/mlbench-core/tree/v2.4.0) (2020-04-20)

[Full Changelog](https://github.com/mlbench/mlbench-core/compare/v2.3.2…v2.4.0)

Implemented enhancements:

• Switch to black for code formatting [#35](https://github.com/mlbench/mlbench-core/issues/35)

Closed issues:

• Travis tests run only for Python 3.6 [#65](https://github.com/mlbench/mlbench-core/issues/65)

• Downloading results fails if –output option is not provided [#57](https://github.com/mlbench/mlbench-core/issues/57)

• Remember user input in mlbench run [#56](https://github.com/mlbench/mlbench-core/issues/56)

• Aggregate the gradients by model, instead of by layers. [#45](https://github.com/mlbench/mlbench-core/issues/45)

• Update docker images to CUDA10, mlbench-core module to newest [#43](https://github.com/mlbench/mlbench-core/issues/43)

• Upgrade PyTorch to 1.4 [#40](https://github.com/mlbench/mlbench-core/issues/40)

Merged pull requests:

• Pytorch v1.4.0 [#68](https://github.com/mlbench/mlbench-core/pull/68) ([ehoelzl](https://github.com/ehoelzl))

• Fix ci [#67](https://github.com/mlbench/mlbench-core/pull/67) ([ehoelzl](https://github.com/ehoelzl))

• Add aggregation by model [#61](https://github.com/mlbench/mlbench-core/pull/61) ([ehoelzl](https://github.com/ehoelzl))

• Remember user input in mlbench run [#60](https://github.com/mlbench/mlbench-core/pull/60) ([mmilenkoski](https://github.com/mmilenkoski))

• Add default name of output file in CLI [#58](https://github.com/mlbench/mlbench-core/pull/58) ([mmilenkoski](https://github.com/mmilenkoski))

• Cli adaptation [#55](https://github.com/mlbench/mlbench-core/pull/55) ([ehoelzl](https://github.com/ehoelzl))

• Update tags and patch version to 2.3.2 [#52](https://github.com/mlbench/mlbench-core/pull/52) ([ehoelzl](https://github.com/ehoelzl))

• Add get_optimizer to create optimizer object [#48](https://github.com/mlbench/mlbench-core/pull/48) ([mmilenkoski](https://github.com/mmilenkoski))

# Changelog

## [v2.3.2](https://github.com/mlbench/mlbench-core/tree/v2.3.2) (2020-04-07)

[Full Changelog](https://github.com/mlbench/mlbench-core/compare/v2.3.1…v2.3.2)

Implemented enhancements:

• Add NCCL & GLOO Backend support [#49](https://github.com/mlbench/mlbench-core/issues/49)

• Add NCCL & GLOO Backend support [#47](https://github.com/mlbench/mlbench-core/pull/47) ([giorgiosav](https://github.com/giorgiosav))

Fixed bugs:

• math ValueError with 1-node cluster [#38](https://github.com/mlbench/mlbench-core/issues/38)

Merged pull requests:

• num_workers fix [#51](https://github.com/mlbench/mlbench-core/pull/51) ([giorgiosav](https://github.com/giorgiosav))

• Adds centralized Adam implementation [#41](https://github.com/mlbench/mlbench-core/pull/41) ([mmilenkoski](https://github.com/mmilenkoski))

# Change Log

## [2.3.1](https://github.com/mlbench/mlbench-core/tree/2.3.1) (2020-03-09) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v2.3.0…2.3.1)

Implemented enhancements:

• Customize Communication Scheme For Sparsified/Quantizatized/Decentralized scenarios [#12](https://github.com/mlbench/mlbench-core/issues/12)

## [v2.3.0](https://github.com/mlbench/mlbench-core/tree/v2.3.0) (2019-12-23) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v2.2.1…v2.3.0)

## [v2.2.1](https://github.com/mlbench/mlbench-core/tree/v2.2.1) (2019-12-16) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v2.2.0…v2.2.1)

# Change Log

## [v2.2.0](https://github.com/mlbench/mlbench-core/tree/v2.2.0) (2019-11-11) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v2.1.0…v2.1.1)

Implemented enhancements: - initialize_backends can now be called as context manager - Improved CLI to run multiple runs in parallel

## [v2.1.1](https://github.com/mlbench/mlbench-core/tree/v2.1.1) (2019-11-11) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v2.1.0…v2.1.1)

## [v2.1.0](https://github.com/mlbench/mlbench-core/tree/v2.1.0) (2019-11-4) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v2.0.0…v2.1.0)

Implemented enhancements:

• Added CLI for MLBench runs

## [v1.4.4](https://github.com/mlbench/mlbench-core/tree/v1.4.4) (2019-05-28) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.4.3…v1.4.4)

## [v1.4.3](https://github.com/mlbench/mlbench-core/tree/v1.4.3) (2019-05-23) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.4.2…v1.4.3)

## [v1.4.2](https://github.com/mlbench/mlbench-core/tree/v1.4.2) (2019-05-21) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.4.1…v1.4.2)

## [v1.4.1](https://github.com/mlbench/mlbench-core/tree/v1.4.1) (2019-05-16) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.4.0…v1.4.1)

## [v1.4.0](https://github.com/mlbench/mlbench-core/tree/v1.4.0) (2019-05-02) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.3.4…v1.4.0)

Implemented enhancements:

• Split Train and Validation in Tensorflow [#22](https://github.com/mlbench/mlbench-core/issues/22)

## [v1.3.4](https://github.com/mlbench/mlbench-core/tree/v1.3.4) (2019-03-20) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.3.3…v1.3.4)

Implemented enhancements:

• in controlflow, don’t mix train and validation [#20](https://github.com/mlbench/mlbench-core/issues/20)

Fixed bugs:

• Add metrics logging for Tensorflow [#19](https://github.com/mlbench/mlbench-core/issues/19)

## [v1.3.3](https://github.com/mlbench/mlbench-core/tree/v1.3.3) (2019-02-26) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.3.2…v1.3.3)

## [v1.3.2](https://github.com/mlbench/mlbench-core/tree/v1.3.2) (2019-02-13) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.3.1…v1.3.2)

## [v1.3.1](https://github.com/mlbench/mlbench-core/tree/v1.3.1) (2019-02-13) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.3.0…v1.3.1)

## [v1.3.0](https://github.com/mlbench/mlbench-core/tree/v1.3.0) (2019-02-12) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.2.1…v1.3.0)

## [v1.2.1](https://github.com/mlbench/mlbench-core/tree/v1.2.1) (2019-01-31) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.2.0…v1.2.1)

## [v1.2.0](https://github.com/mlbench/mlbench-core/tree/v1.2.0) (2019-01-30) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.1.1…v1.2.0)

## [v1.1.1](https://github.com/mlbench/mlbench-core/tree/v1.1.1) (2019-01-09) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.1.0…v1.1.1)

## [v1.1.0](https://github.com/mlbench/mlbench-core/tree/v1.1.0) (2018-12-06) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.0.0…v1.1.0)

Fixed bugs:

• Bug when saving checkpoints [#13](https://github.com/mlbench/mlbench-core/issues/13)

## [v1.0.0](https://github.com/mlbench/mlbench-core/tree/v1.0.0) (2018-11-20) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/1.4.2…v1.0.0)

Implemented enhancements:

• Add API Client to mlbench-core [#6](https://github.com/mlbench/mlbench-core/issues/6)

• Move to google-style docs [#4](https://github.com/mlbench/mlbench-core/issues/4)

• Add Imagenet Dataset for pytorch [#3](https://github.com/mlbench/mlbench-core/issues/3)

• Move worker code to mlbench-core repo [#1](https://github.com/mlbench/mlbench-core/issues/1)

# Change Log

## [1.4.2](https://github.com/mlbench/mlbench-core/tree/1.4.2) (2019-05-21) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.0.0…1.4.2)

Implemented enhancements:

• Split Train and Validation in Tensorflow [#22](https://github.com/mlbench/mlbench-core/issues/22)

• in controlflow, don’t mix train and validation [#20](https://github.com/mlbench/mlbench-core/issues/20)

Fixed bugs:

• Add metrics logging for Tensorflow [#19](https://github.com/mlbench/mlbench-core/issues/19)

• Bug when saving checkpoints [#13](https://github.com/mlbench/mlbench-core/issues/13)

# Change Log

## [v1.4.1](https://github.com/mlbench/mlbench-core/tree/v1.4.1) (2019-05-16) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.4.0…v1.4.1)

## [1.4.0](https://github.com/mlbench/mlbench-core/tree/1.4.0) (2019-05-02) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.0.0…1.4.0)

Implemented enhancements:

• Split Train and Validation in Tensorflow [#22](https://github.com/mlbench/mlbench-core/issues/22)

• in controlflow, don’t mix train and validation [#20](https://github.com/mlbench/mlbench-core/issues/20)

Fixed bugs:

• Add metrics logging for Tensorflow [#19](https://github.com/mlbench/mlbench-core/issues/19)

• Bug when saving checkpoints [#13](https://github.com/mlbench/mlbench-core/issues/13)

# Change Log

## [v1.3.4](https://github.com/mlbench/mlbench-core/tree/v1.3.4) (2019-03-20) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.3.3…v1.3.4)

Implemented enhancements:

• in controlflow, don’t mix train and validation [#20](https://github.com/mlbench/mlbench-core/issues/20)

Fixed bugs:

• Add metrics logging for Tensorflow [#19](https://github.com/mlbench/mlbench-core/issues/19)

## [v1.3.3](https://github.com/mlbench/mlbench-core/tree/v1.3.3) (2019-02-26) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.3.2…v1.3.3)

## [v1.3.2](https://github.com/mlbench/mlbench-core/tree/v1.3.2) (2019-02-13) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.3.1…v1.3.2)

## [v1.3.1](https://github.com/mlbench/mlbench-core/tree/v1.3.1) (2019-02-13) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.3.0…v1.3.1)

## [v1.3.0](https://github.com/mlbench/mlbench-core/tree/v1.3.0) (2019-02-12) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.2.1…v1.3.0)

## [v1.2.1](https://github.com/mlbench/mlbench-core/tree/v1.2.1) (2019-01-31) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.2.0…v1.2.1)

## [v1.2.0](https://github.com/mlbench/mlbench-core/tree/v1.2.0) (2019-01-30) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.1.1…v1.2.0)

## [v1.1.1](https://github.com/mlbench/mlbench-core/tree/v1.1.1) (2019-01-09) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.1.0…v1.1.1)

# Change Log

## [v1.1.0](https://github.com/mlbench/mlbench-core/tree/v1.1.0) (2018-12-06) [Full Changelog](https://github.com/mlbench/mlbench-core/compare/v1.0.0…v1.1.0)

Fixed bugs:

• Bug when saving checkpoints [#13](https://github.com/mlbench/mlbench-core/issues/13)

• Adds Tensorflow Controlflow, Dataset and Model code

• Adds Pytorch linear models

• Adds sparsified and decentralized optimizers

## [v1.0.0](https://github.com/mlbench/mlbench-core/tree/v1.0.0) (2018-11-15)

Implemented enhancements:

• Add API Client to mlbench-core [#6](https://github.com/mlbench/mlbench-core/issues/6)

• Move to google-style docs [#4](https://github.com/mlbench/mlbench-core/issues/4)

• Add Imagenet Dataset for pytorch [#3](https://github.com/mlbench/mlbench-core/issues/3)

• Move worker code to mlbench-core repo [#1](https://github.com/mlbench/mlbench-core/issues/1)

## [0.1.0](https://github.com/mlbench/mlbench/tree/0.1.0) (2018-09-14) Implemented enhancements:

• Add documentation in reference implementation to docs [#46](https://github.com/mlbench/mlbench/issues/46)

• Replace cAdvisor with Kubernetes stats for Resource usage [#38](https://github.com/mlbench/mlbench/issues/38)

• Rename folders [#31](https://github.com/mlbench/mlbench/issues/31)

• Change docker image names [#30](https://github.com/mlbench/mlbench/issues/30)

• Add continuous output for mpirun [#27](https://github.com/mlbench/mlbench/issues/27)

• Replace SQlite with Postgres [#25](https://github.com/mlbench/mlbench/issues/25)

• Fix unittest [#23](https://github.com/mlbench/mlbench/issues/23)

• Add/Fix CI/Automated build [#22](https://github.com/mlbench/mlbench/issues/22)

• Cleanup unneeded project files [#21](https://github.com/mlbench/mlbench/issues/21)

• Remove hardcoded values [#20](https://github.com/mlbench/mlbench/issues/20)

• Improves Notes.txt [#19](https://github.com/mlbench/mlbench/issues/19)

• Rename components [#15](https://github.com/mlbench/mlbench/issues/15)

Fixed bugs:

• 504 Error when downloading metrics for long runs [#61](https://github.com/mlbench/mlbench/issues/61)

Closed issues:

• small doc improvements for first release [#54](https://github.com/mlbench/mlbench/issues/54)

• Check mlbench works on Google Cloud [#51](https://github.com/mlbench/mlbench/issues/51)

• learning rate scheduler [#50](https://github.com/mlbench/mlbench/issues/50)

• Add Nvidia k8s-device-plugin to charts [#48](https://github.com/mlbench/mlbench/issues/48)

• Add Weave to Helm Chart [#41](https://github.com/mlbench/mlbench/issues/41)

• Allow limiting of resources for experiments [#39](https://github.com/mlbench/mlbench/issues/39)

• Allow downloading of Run measurements [#35](https://github.com/mlbench/mlbench/issues/35)

• Worker Details page [#33](https://github.com/mlbench/mlbench/issues/33)

• Run Visualizations [#32](https://github.com/mlbench/mlbench/issues/32)

• Show experiment history in Dashboard [#18](https://github.com/mlbench/mlbench/issues/18)

• Show model progress in Dashboard [#13](https://github.com/mlbench/mlbench/issues/13)

• Report cluster status in Dashboard [#12](https://github.com/mlbench/mlbench/issues/12)

• Send metrics from SGD example to metrics api [#11](https://github.com/mlbench/mlbench/issues/11)

• Add metrics endpoint for experiments [#10](https://github.com/mlbench/mlbench/issues/10)

• Let Coordinator Dashboard start a distributed Experiment [#9](https://github.com/mlbench/mlbench/issues/9)

• Add mini-batch SGD model experiment [#8](https://github.com/mlbench/mlbench/issues/8)

• add benchmark code for MPI [#7](https://github.com/mlbench/mlbench/issues/7)

• add benchmark code for tensorflow [#6](https://github.com/mlbench/mlbench/issues/6)

• add benchmark code for apache reef [#5](https://github.com/mlbench/mlbench/issues/5)

• add benchmark code for apache flink [#4](https://github.com/mlbench/mlbench/issues/4)

• get initial benchmark numbers (spark reference implementation and mllib/ml) [#3](https://github.com/mlbench/mlbench/issues/3)

• evaluate script (framework-independent) and algorithm output format [#2](https://github.com/mlbench/mlbench/issues/2)

• bench-spark: remove prepare-data for now, comment on solver prequisites [#1](https://github.com/mlbench/mlbench/issues/1)

* This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)

* This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)

* This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)

* This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)

* This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)

* This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)

* This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)

* This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)

* This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)

* This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)

* This Changelog was automatically generated by [github_changelog_generator](https://github.com/github-changelog-generator/github-changelog-generator)

* This Changelog was automatically generated by [github_changelog_generator](https://github.com/github-changelog-generator/github-changelog-generator)