Skip to main content

A public and reproducible collection of reference implementations and benchmark suite for distributed machine learning systems.

Project description

====================================================
mlbench_core: Distributed Machine Learning Benchmark
====================================================

.. image:: https://travis-ci.com/mlbench/mlbench.svg?branch=develop
:target: https://travis-ci.com/mlbench/mlbench

.. image:: https://readthedocs.org/projects/mlbench/badge/?version=latest
:target: https://mlbench.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status




A public and reproducible collection of reference implementations and benchmark suite for distributed machine learning algorithms, frameworks and systems.


* Project website: https://mlbench.github.io/
* Free software: Apache Software License 2.0
* Documentation: https://mlbench.readthedocs.io.


Features
--------

* For reproducibility and simplicity, we currently focus on standard **supervised ML**, including standard deep learning tasks as well as classic linear ML models.
* We provide **reference implementations** for each algorithm, to make it easy to port to a new framework.
* Our goal is to benchmark all/most currently relevant **distributed execution frameworks**. We welcome contributions of new frameworks in the benchmark suite.
* We provide **precisely defined tasks** and datasets to have a fair and precise comparison of all algorithms, frameworks and hardware.
* Independently of all solver implementations, we provide universal **evaluation code** allowing to compare the result metrics of different solvers and frameworks.
* Our benchmark code is easy to run on **public clouds**.



Community
---------

About us: See :doc:`Authors </authors>`

Mailing list: https://groups.google.com/d/forum/mlbench

Contact Email: mlbench-contact@googlegroups.com


# Change Log

## [1.0.0](https://github.com/mlbench/mlbench-core/tree/1.0.0) (2018-11-15)

**Implemented enhancements:**

- Add API Client to mlbench-core [\#6](https://github.com/mlbench/mlbench-core/issues/6)
- Move to google-style docs [\#4](https://github.com/mlbench/mlbench-core/issues/4)
- Add Imagenet Dataset for pytorch [\#3](https://github.com/mlbench/mlbench-core/issues/3)
- Move worker code to mlbench-core repo [\#1](https://github.com/mlbench/mlbench-core/issues/1)

# Change Log

## [0.1.0](https://github.com/mlbench/mlbench/tree/0.1.0) (2018-09-14)
**Implemented enhancements:**

- Add documentation in reference implementation to docs [\#46](https://github.com/mlbench/mlbench/issues/46)
- Replace cAdvisor with Kubernetes stats for Resource usage [\#38](https://github.com/mlbench/mlbench/issues/38)
- Rename folders [\#31](https://github.com/mlbench/mlbench/issues/31)
- Change docker image names [\#30](https://github.com/mlbench/mlbench/issues/30)
- Add continuous output for mpirun [\#27](https://github.com/mlbench/mlbench/issues/27)
- Replace SQlite with Postgres [\#25](https://github.com/mlbench/mlbench/issues/25)
- Fix unittest [\#23](https://github.com/mlbench/mlbench/issues/23)
- Add/Fix CI/Automated build [\#22](https://github.com/mlbench/mlbench/issues/22)
- Cleanup unneeded project files [\#21](https://github.com/mlbench/mlbench/issues/21)
- Remove hardcoded values [\#20](https://github.com/mlbench/mlbench/issues/20)
- Improves Notes.txt [\#19](https://github.com/mlbench/mlbench/issues/19)
- Rename components [\#15](https://github.com/mlbench/mlbench/issues/15)

**Fixed bugs:**

- 504 Error when downloading metrics for long runs [\#61](https://github.com/mlbench/mlbench/issues/61)

**Closed issues:**

- small doc improvements for first release [\#54](https://github.com/mlbench/mlbench/issues/54)
- Check mlbench works on Google Cloud [\#51](https://github.com/mlbench/mlbench/issues/51)
- learning rate scheduler [\#50](https://github.com/mlbench/mlbench/issues/50)
- Add Nvidia k8s-device-plugin to charts [\#48](https://github.com/mlbench/mlbench/issues/48)
- Add Weave to Helm Chart [\#41](https://github.com/mlbench/mlbench/issues/41)
- Allow limiting of resources for experiments [\#39](https://github.com/mlbench/mlbench/issues/39)
- Allow downloading of Run measurements [\#35](https://github.com/mlbench/mlbench/issues/35)
- Worker Details page [\#33](https://github.com/mlbench/mlbench/issues/33)
- Run Visualizations [\#32](https://github.com/mlbench/mlbench/issues/32)
- Show experiment history in Dashboard [\#18](https://github.com/mlbench/mlbench/issues/18)
- Show model progress in Dashboard [\#13](https://github.com/mlbench/mlbench/issues/13)
- Report cluster status in Dashboard [\#12](https://github.com/mlbench/mlbench/issues/12)
- Send metrics from SGD example to metrics api [\#11](https://github.com/mlbench/mlbench/issues/11)
- Add metrics endpoint for experiments [\#10](https://github.com/mlbench/mlbench/issues/10)
- Let Coordinator Dashboard start a distributed Experiment [\#9](https://github.com/mlbench/mlbench/issues/9)
- Add mini-batch SGD model experiment [\#8](https://github.com/mlbench/mlbench/issues/8)
- add benchmark code for MPI [\#7](https://github.com/mlbench/mlbench/issues/7)
- add benchmark code for tensorflow [\#6](https://github.com/mlbench/mlbench/issues/6)
- add benchmark code for apache reef [\#5](https://github.com/mlbench/mlbench/issues/5)
- add benchmark code for apache flink [\#4](https://github.com/mlbench/mlbench/issues/4)
- get initial benchmark numbers \(spark reference implementation and mllib/ml\) [\#3](https://github.com/mlbench/mlbench/issues/3)
- evaluate script \(framework-independent\) and algorithm output format [\#2](https://github.com/mlbench/mlbench/issues/2)
- bench-spark: remove prepare-data for now, comment on solver prequisites [\#1](https://github.com/mlbench/mlbench/issues/1)



\* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)*

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mlbench_core-1.0.0.tar.gz (41.3 kB view hashes)

Uploaded Source

Built Distribution

mlbench_core-1.0.0-py3-none-any.whl (75.0 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page