mlf-core 1.9.0

Reproducible machine learning pipelines using mlflow.

.. image:: https://user-images.githubusercontent.com/21954664/84388841-84b4cc80-abf5-11ea-83f3-b8ce8de36e25.png
:target: https://mlf-core.com
:alt: mlf-core logo

|

========
mlf-core
========

.. image:: https://github.com/mlf-core/mlf-core/workflows/Build%20mlf-core%20Package/badge.svg?branch=master
:target: https://github.com/mlf-core/mlf-core/actions?query=workflow%3A%22Build+mlf-core+Package%22
:alt: Github Workflow Build mlf-core Status

.. image:: https://github.com/mlf-core/mlf-core/workflows/Run%20mlf-core%20Tox%20Test%20Suite/badge.svg?branch=master
:target: https://github.com/mlf-core/mlf-core/actions?query=workflow%3A%22Run+mlf-core+Tox+Test+Suite%22
:alt: Github Workflow Tests Status

.. image:: https://img.shields.io/pypi/v/mlf-core.svg
:target: https://pypi.python.org/pypi/mlf-core
:alt: PyPI Status

.. image:: https://static.pepy.tech/personalized-badge/mlf-core?units=international_system&left_color=grey&right_color=green&left_text=Downloads
:target: https://pepy.tech/project/mlf-core
:alt: Pepy Downloads

.. image:: https://img.shields.io/discord/742367395196305489?color=passing
:target: https://discord.gg/Mv8sAcq
:alt: Discord

.. image:: https://readthedocs.org/projects/mlf-core/badge/?version=latest
:target: https://mlf-core.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status

.. image:: https://flat.badgen.net/dependabot/thepracticaldev/dev.to?icon=dependabot
:target: https://flat.badgen.net/dependabot/thepracticaldev/dev.to?icon=dependabot
:alt: Dependabot Enabled


Fully GPU deterministic machine learning project templates using MLflow_.

* Free software: Apache2.0
* Documentation: https://mlf-core.readthedocs.io.


Features
--------

* Jumpstart your machine learning project with fully fledged, (multi) GPU deterministic mlflow project templates
* Pytorch, Tensorflow, XGBoost supported
* Conda and Docker support out of the box

Installing
---------------

Start your journey with mlf-core by installing it via ``$ pip install mlf-core``.

See `Installation <https://mlf_core.readthedocs.io/en/latest/readme.html#installing>`_.

config
------
Configure mlf-core to get started.

.. figure:: https://user-images.githubusercontent.com/31141763/102669098-f6199d00-418d-11eb-9ae6-26c12d9c1231.gif

See `Configuring mlf-core <https://mlf_core.readthedocs.io/en/latest/config.html>`_

list
----
List all available mlf-core templates.

.. figure:: https://user-images.githubusercontent.com/31141763/102668939-8d322500-418d-11eb-8b2c-acd895fc50e3.gif

See `Listing all templates <https://mlf_core.readthedocs.io/en/latest/list_info.html#list>`_.

info
----
Get detailed information on a mlf-core template.

.. figure:: https://user-images.githubusercontent.com/31141763/102669191-324cfd80-418e-11eb-9542-d2995b7318a9.gif

See `Get detailed template information <https://mlf_core.readthedocs.io/en/latest/list_info.html#info>`_.

create
------
Kickstart your deterministic machine laerning project with one of mlf-core's templates in no time.

.. figure:: https://user-images.githubusercontent.com/31141763/102669143-1184a800-418e-11eb-853b-0deb0387efc6.gif

See `Create a project <https://mlf_core.readthedocs.io/en/latest/create.html>`_.

lint
----
Use advanced linting to ensure your project always adheres to mlf-core's standards and stays deterministic.

.. image:: https://user-images.githubusercontent.com/31141763/102668893-696edf00-418d-11eb-888e-822244a6f5dc.gif

See `Linting your project <https://mlf_core.readthedocs.io/en/latest/lint.html>`_

bump-version
------------
Bump your project version across several files.

.. figure:: https://user-images.githubusercontent.com/31141763/102668987-aaff8a00-418d-11eb-9292-dc512f77f09b.gif

See `Bumping the version of an existing project <https://mlf_core.readthedocs.io/en/latest/bump_version.html>`_.

sync
------
Sync your project with the latest mlf-core release to get the latest template features.

.. figure:: https://user-images.githubusercontent.com/31141763/102669065-de421900-418d-11eb-9e1b-a76487d02b2a.gif

See `Syncing a project <https://mlf_core.readthedocs.io/en/latest/sync.html>`_.

upgrade
-------
Check whether you are using the latest mlf-core version and update automatically to benefit from the latest features.

See `<https://mlf_core.readthedocs.io/en/latest/upgrade.html>`_.


.. figure:: https://user-images.githubusercontent.com/21954664/98472352-c2dd0900-21f2-11eb-9fe3-929b2a21bd4c.png
:scale: 100 %
:alt: mlf-core summary

mlf-core enables deterministic machine learning. MLflow and a provided Read the Docs setup ensure that all hyperparameters, metrics and model details are well documented.
Reproducible environments are provided with the use of Conda and Docker. Finally, the mlf-core ecosystem ensures that all library specific settings required for determinism are enabled,
no non-deterministic algorithms are used and that the used hardware is tracked.

Credits
-------

Primary idea and main development by `Lukas Heumos <https://github.com/zethson/>`_. mlf-core is inspired by nf-core_.
This package was created with cookietemple_ based on a modified `audreyr/cookiecutter-pypackage`_ project template using cookiecutter_.

.. _MLflow: https://mlflow.org
.. _cookietemple: https://cookietemple.com
.. _cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage
.. _nf-core: https://nf-co.re


.. _changelog_f:

==========
Changelog
==========

This project adheres to `Semantic Versioning <https://semver.org/>`_.

1.9.0 (2021-02-16)
------------------

**Added**

* Possibility to log input files in all template with the MLFCore object.
* [ALL TEMPLATES] Using new mlf-core/base:1.2.0 container, which is based on CUDA 11.2.1 and cudnn 8.1
* [PYTORCH] Upgraded Pytorch to 1.7.1
* [PYTORCH] Added set_deterministic
* [ALL TEMPLATES] Using new mlflow autolog
* [ALL TEMPLATES] Changed mlflow autolog for loss to every 1 iteration

**Fixed**

* mlf-core fix-artifact-paths does now operate as expected.
* [ALL TEMPLATES] fixed a path error that causes the general template linter to fail searching for
subprocess.call([\'conda\', \'env\', \'export\', \'--name\', \'<<project_name>>\'], stdout=conda_env_filehandler) and
mlflow.log_artifact(f\'{{reports_output_dir}}/<<project_name>>_conda_environment.yml\', artifact_path=\'reports\') in the project's mlf_core.py file

**Dependencies**

**Deprecated**


1.8.0 (2021-02-01)
------------------

**Added**

* fixed sync command: blacklisted files now fully working
* updated sync code to the latest version and added logging
* added a output-directory parameter option to create command, allowing users to specifify the
directory, where the project should be created
* added a user information before the user is prompted to create a GitHub repo with mlf-core
* added flake8 linting for each create template workflow
* GitHub workflow badges in README now link to the corresponding workflow
* [ALL TEMPLATES] added vscode files to .gitignore for templates
* [ALL TEMPLATES] now feature a mlf-core lint workflow with colored linting result output
* [ALL TEMPLATES] renamed train_cpu workflows to use project_slug (was project_slug_no_hyphen)
* [MLFLOW TEMPLATES] refactored the common files for all mlflow templates into a common files directory
* config default values are now preconfigured values (if any)
* added logging to all commands mlf-core offers
* refactored sync, bump-version and create code (added type hints, fixed some variable scoping)
* [ALL TEMPLATES] replaced click with argparse
* [ALL TEMPLATES] renamed some parameters to harmonize between templates
* [PyTorch Template] New template with autologging via pytorch-lightning and mlflow 1.13.1

**Fixed**

* fixed publish_docs WF for all templates working on main or master branch

* fixed gh_pages setup for default branch main

* fixed a bug causing the check upgrade version function to fail if local version is a SNAPSHOT version

**Dependencies**

**Deprecated**


1.7.8 (2020-12-04)
------------------

**Added**

* Instructions to make Docker container public

**Fixed**

**Dependencies**

**Deprecated**


1.7.7 (2020-11-29)
------------------

**Added**

* Support for deploying the documentation on Github Pages. By default the Documentation is pushed to the gh-pages branch.
Simply enable Github pages (repository settings) with the gh-pages branch and your documentation will build on ``https://username.github.io/repositoryname``

**Fixed**

* Workflows are now also triggered on PR

**Dependencies**

**Deprecated**


1.7.6 (2020-11-22)
------------------

**Added**

**Fixed**

* Github project creation support due to Github's new main branch

**Dependencies**

**Deprecated**

1.7.5 (2020-11-18)
------------------

**Added**

**Fixed**

sync workflow set-env

**Dependencies**

**Deprecated**


1.7.4 (2020-11-11)
------------------

**Added**

**Fixed**

* Sync now compares against the development branch and not the master branch.

**Dependencies**

**Deprecated**


1.7.3 (2020-11-09)
------------------

**Added**

**Fixed**

* Added CHANGELOG.rst to blacklisted files

**Dependencies**

**Deprecated**


1.7.2 (2020-11-07)
------------------

**Added**

**Fixed**

* Removed redundant print in xgboost

**Dependencies**

**Deprecated**


1.7.1 (2020-11-07)
------------------

**Added**

**Fixed**

* mlf-core sync does now correctly find attributes

**Dependencies**

**Deprecated**


1.7.0 (2020-11-06)
------------------

**Added**

* fix-artifact-paths which replaces the artifact paths with the paths of the current system
* More structured documentation

**Fixed**

* Now using GPUs by default only when GPUs are available for XGBoost templates

**Dependencies**

**Deprecated**


1.6.1 (2020-11-06)
------------------

**Added**

* Workflows for package-prediction
* Documentation for package-prediction

**Fixed**

**Dependencies**

**Deprecated**


1.6.0 (2020-11-02)
------------------

**Added**

* New package templates (package-prediction) for Pytorch, Tensorflow and XGBoost

**Fixed**

**Dependencies**

**Deprecated**


1.5.0 (2020-10-29)
------------------

**Added**

* Check for non-deterministic functions for mlflow-tensorflow linter
* Check for all_reduce for mlflow-xgboost templates
* Check for OS for system-intelligence runs. If not Linux -> don't run system-intelligence
* .gitattributes to templates, which ignores mlruns
* Documentation on creating releases

**Fixed**

* Sync now operates correctly with the correct PR URL

**Dependencies**

**Deprecated**


1.4.4 (2020-10-22)
------------------

**Added**

**Fixed**

* Conda report generation

**Dependencies**

**Deprecated**


1.4.3 (2020-09-17)
------------------

**Added**

**Fixed**

* Internal Github workflows
* Docker documentation

**Dependencies**

**Deprecated**

1.4.2 (2020-09-11)
------------------

**Added**

**Fixed**

* Accidentally left a - in the train_cpu.yml of mlflow-pytorch
* mlflow-pytorch and mlflow-tensorflow now only train for 2 epochs on train_cpu.yml

**Dependencies**

**Deprecated**


1.4.1 (2020-09-10)
------------------

**Added**

**Fixed**

* Github username must now always be lowercase, since Docker does not like uppercase letters
* Fixed train_cpu workflows to use the correct containers

**Dependencies**

**Deprecated**

1.4.0 (2020-08-28)
------------------

**Added**

* model.rst documentation for all templates
* added support for verbose output

**Fixed**

* Publish Docker workflows now use the new Github registry
* Default Docker container names are now ```image: ghcr.io/{{ cookiecutter.github_username }}/{{ cookiecutter.project_slug_no_hyphen }}:{{ cookiecutter.version }}```

**Dependencies**

**Deprecated**


1.3.0 (2020-08-27)
------------------

**Added**

* automatically mounting /data now in all mlflow templates (#56)
* mlflow-xgboost xgboost from 1.1.1 to 1.2.0

**Fixed**

* mlf_core.py now uses project_slug; adapted linter accordingly (#55)
* Removed dask-cuda from mlflow-xgboost

**Dependencies**

**Deprecated**


1.2.2 (2020-08-21)
------------------

**Added**

**Fixed**

* A couple of parameters were not with hyphen -> now default behavior

**Dependencies**

**Deprecated**


1.2.1 (2020-08-21)
------------------

**Added**

**Fixed**

* flake8 for mlflow-pytorch

**Dependencies**

**Deprecated**


1.2.0 (2020-08-21)
------------------

**Added**

* Option --view to config to view the current configuration
* Option --set_token to sync to set the sync token again

**Fixed**

* #41 https://github.com/mlf-core/mlf-core/issues/41 -> mlflow-pytorch multi GPU Support

**Dependencies**

**Deprecated**


1.1.0 (2020-08-19)
------------------

**Added**

* Publish Docker workflow. Publishes to Github Packages per default, but can be configured.
* Linting function, which checks mlflow-pytorch for any used atomic_add functions.
* system-intelligence 1.2.2 -> 1.2.3
* Support for both, MLF-CORE TODO: and TODO MLF-CORE: statements

**Fixed**

* Default project version from 0.1.0 to 0.1.0-SNAPSHOT.
* Outdated screenshots
* Nightly versions now warn instead of wrongly complaining about outdated versions.
* Sync actor, but not yet completely for organizations
* A LOT of documentation
* Now using project_slug_no_hyphen to facilitate the creation of repositories with - characters.
* Removed boston dataset from XGBoost and XGBoost_dask
* Renamed all parameters to use hyphens instead of underscores

**Dependencies**

**Deprecated**


1.0.1 (2020-08-11)
------------------

**Added**

**Fixed**

* Sync workflow now uses the correct secret

**Dependencies**

**Deprecated**


1.0.0 (2020-08-11)
------------------

**Added**

* Created the project using cookietemple
* Added all major commands: create, list, info, lint, sync, bump-version, config, upgrade
* Added mlflow-pytorch, mlflow-tensorflow, mlflow-xgboost, mlflow-xgboost_dask templates

**Fixed**

**Dependencies**

**Deprecated**