🔗 Important links: Documentation | Issue Tracker | Changelog | License

This software package provides mlrl-testbed - a command line utility for running machine learning experiments. It implements a straightforward, easily configurable, and extensible workflow for conducting experiments.

mlrl-testbed

On its own, this package is not very powerful. It is intended as a basis for other packages that build functionality upon it. In fact, it does not make any assumptions about the problem domain or type of machine learning algorithm that should be used in an experiment. Instead, implementations of domain- or algorithm-specific functionality are provided by the extensions discussed below.

Tabular Machine Learning

The package mlrl-testbed-sklearn adds support for tabular machine learning problems by making use of the scikit-learn framework. It can easily be installed via the following command (and will pull mlrl-testbed as a dependency):

pip install mlrl-testbed-sklearn

💡 Example 1: Running an Experiment

After installing the package the machine learning algorithms offered by the scikit-learn project can be configured and experimented with via a command line API. For example, the following command runs an experiment using scikit-learn's RandomForestClassifier:

mlrl-testbed mlrl.testbed_sklearn \
    --data-dir path/to/datasets/ \
    --dataset dataset-name \
    --estimator RandomForestClassifier
    --n-estimators 50

The above command does not only train a model, but also evaluates it according to common measures and prints the evaluation results. It also demonstrates how algorithmic parameters, like --n-estimator, can be controlled via command line arguments. A description of all available arguments can be obtained by specifying the --help flag.

💡 Example 2: Using a Meta-Estimator

By using the argument --meta-estimator, it is also possible to use so-called "meta-estimators" in experiments. These are algorithms that decompose the given problem in some way and make use of a "base estimator" to solve the subproblems individually. The following example demonstrates how the meta-estimator ClassifierChain can be used with the base estimator DecisionTreeClassifier:

mlrl-testbed mlrl.testbed_sklearn \
    --data-dir path/to/datasets/ \
    --dataset dataset-name \
    --meta-estimator ClassifierChain \
    --estimator DecisionTreeClassifier \
    --max-depth 5 \
    --meta-verbose true

Whereas the argument --max-depth belongs to the base learner (and controls the maximum depths of individual trees), the argument --meta-verbose belongs to the meta-estimator (and instructs it to produce more verbose log output). All arguments associated with meta-estimators start with the "meta" prefix.

💡 Example 3: Running Multiple Experiments

It is also possible to run multiple experiments at once by defining the datasets and algorithmic parameters to be used in the different runs in a YAML file (if the package mlrl-testbed-slurm is installed, individual experiments can be submitted to a SLURM cluster by appending the argument --runner slurm):

mlrl-testbed mlrl.testbed_sklearn \
    --mode batch \
    --estimator RandomForestClassifier \
    --config path/to/config.yaml \
    --base-dir path/to/results
    --store-evaluation true

An exemplary YAML file is shown below. Each combination of the specified parameter values is applied to each dataset defined in the file.

datasets:
  - directory: path/to/datasets/
    names:
      - first-dataset
      - second-dataset
parameters:
  - name: --n-estimators
    values:
      - 50
      - 100

The experimental results produced before, can afterward be viewed by using read mode. The output of the following command does only include results for individual experiments, but also aggregates them into a condensed overview:

mlrl-testbed mlrl.testbed_sklearn \
    --mode read \
    --input-dir path/to/results \
    --print-evaluation true

🏁 Advantages

Making use of mlrl-testbed does not only help with the burdens of training and evaluating machine learning models, it can also help making your own methods and algorithms more accessible to users. This is demonstrated by the rule learning algorithms mlrl-boomer and mlrl-seco that can easily be run via the command line API described above and even extend it with rule-specific functionalities.

🔧 Functionalities

The package mlrl-testbed-sklearn provides a command line API that allows configuring and running machine learning algorithms. It allows to apply machine learning algorithms to different datasets and can evaluate their predictive performance in terms of commonly used measures. In detail, it supports the following functionalities:

• Single- and multi-output datasets in the LIBSVM are supported out-of the box. Datasets in the Mulan and MEKA format are supported with the help of the package mlrl-testbed-arff.
• Datasets can automatically be split into training and test data, including the possibility to use cross validation. Alternatively, predefined splits can be provided as separate files.
• One-hot-encoding can be applied to nominal or binary features.
• Binary predictions, scores, or probability estimates can be obtained from machine learning algorithms, if supported. Evaluation measures that are suited for the respective type of predictions are picked automatically.

Furthermore, the command line API provides many options for controlling the experimental results to be gathered during an experiment. Depending on the configuration, the following experimental results can be saved to output files or printed on the console:

• Evaluation scores according to commonly used measures
• Characteristics, i.e., statistical properties, of datasets
• Predictions and their characteristics
• Unique label vectors contained in a classification dataset

If the following are written to output files, they can be loaded and reused in future experiments:

• The machine learning models that have been learned
• Algorithmic parameters used for training

📚 Documentation

Our documentation provides an extensive user guide, as well as an API reference for developers.

• Examples of how to save experimental results to output files.
• Instructions for using your own algorithms with the command line API.
• An overview of available command line arguments for controlling experiments.

For an overview of changes and new features that have been included in past releases, please refer to the changelog.

📜 License

This project is open source software licensed under the terms of the MIT license. We welcome contributions to the project to enhance its functionality and make it more accessible to a broader audience. A frequently updated list of contributors is available here.

All contributions to the project and discussions on the issue tracker are expected to follow the code of conduct.