Skip to main content

Machine Learning Performance Testing Framework

Project description

Build Status MIT license pypi badge

tempeh

tempeh is a framework to

TEst

Machine learning

PErformance

exHaustively

which includes tracking memory usage and run time. This is particularly useful as a pluggable tool for your repository's performance tests. Typically, people want to run them periodically over various datasets and/or with a number of models to catch regressions with respect to run time or memory consumption. This should be as easy as

import pytest
from time import time
from tempeh.configurations import datasets, models

@pytest.mark.parametrize('Dataset', datasets.values())
@pytest.mark.parametrize('Model', models.values())
def test_fit_predict_regression(Dataset, Model):
    dataset = Dataset()
    X_train, X_test = dataset.get_X()
    y_train, y_test = dataset.get_y()
    model = Model()
    max_execution_time = get_max_execution_time(dataset, model)
    if model.compatible_with_dataset(dataset):
        start_time = time()
        model.fit(X_train, y_train)
        model.predict(X_test)
        duration = time() - start_time

        assert duration < max_execution_time

Installation

tempeh depends on various packages to provide models, including tensorflow, torch, xgboost, lightgbm. To install a release version of tempeh just run

pip install tempeh
Common issues
  • If you're using a 32-bit Python version you might need to switch to a 64-bit Python version first to successfully install tensorflow.
  • If the installation of torch fails try using the recommendation from the pytorch website for stable versions without CUDA for your python version on your operating system.
  • If the installation of lightgbm or xgboost fails try to use a pip version less than 20.0 until their bug is resolved.

Structure

Datasets

Datasets (located in the datasets/ directory) encapsulate different datasets used for testing.

To add a new one

  • Create a python file in the datasets/ directory with naming convention [name]_datasets.py
  • Subclass BasePerformanceDatasetWrapper. The naming convention is [dataset_name]PerformanceDatasetWrapper
  • In __init__ load the dataset and call super().__init__(data, targets, size)
  • Add the class to __init__.py
  • Make sure the class contains class variables task, data_type, size
  • Add an entry to the datasets dictionary in configurations.py.

Models

Models (models/ directory) wrap different machine learning models.

To add a new one

  • Create a python file in the models/ directory with naming convention [name]_model.py
  • Subclass BaseModelWrapper and name the class [name]ModelWrapper
  • In __init__ train the model; we expect format __init__(self, ...)
  • Models must contain tasks and algorithm
  • Add an entry to the models dictionary in configurations.py.

Maintainers

In alphabetical order:

Contributing

To contribute please check our Contributing Guide.

Issues

Regular (non-Security) Issues

Please submit a report through Github issues. A maintainer will respond within a reasonable period of time to handle the issue as follows:

  • bug: triage as bug and provide estimated timeline based on severity
  • feature request: triage as feature request and provide estimated timeline
  • question or discussion: triage as question and respond or notify/identify a suitable expert to respond

Maintainers are supposed to link duplicate issues when possible.

Reporting Security Issues

Please take a look at our guidelines for reporting security issues.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tempeh-0.1.12.tar.gz (22.9 kB view details)

Uploaded Source

Built Distribution

tempeh-0.1.12-py3-none-any.whl (39.4 kB view details)

Uploaded Python 3

File details

Details for the file tempeh-0.1.12.tar.gz.

File metadata

  • Download URL: tempeh-0.1.12.tar.gz
  • Upload date:
  • Size: 22.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.4

File hashes

Hashes for tempeh-0.1.12.tar.gz
Algorithm Hash digest
SHA256 9565fbeb2bef5b23f3aa5412fa33956b7df520d294e27d403f1e3a02e3a26b65
MD5 85629780899e8039343572640a29bd60
BLAKE2b-256 761b6cec6ab29489a619141fa0312da7157acebdf2cf9ffe37ecfd4a61365ea1

See more details on using hashes here.

File details

Details for the file tempeh-0.1.12-py3-none-any.whl.

File metadata

  • Download URL: tempeh-0.1.12-py3-none-any.whl
  • Upload date:
  • Size: 39.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.4

File hashes

Hashes for tempeh-0.1.12-py3-none-any.whl
Algorithm Hash digest
SHA256 8bf4d276d09a301ff8c5e105855d03a7d1e89f495b5ad913edaf3d5d16a39e03
MD5 352136e7298e661be498e64de1b7d9d8
BLAKE2b-256 b71bd016f8d2ac250b7df50ffb630c3f71371d35ac92deb80653b0e4d2f9c1a9

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page