Machine Learning Performance Testing Framework
Project description
tempeh
tempeh is a framework to
TEst
Machine learning
PErformance
exHaustively
which includes tracking memory usage and run time. This is particularly useful as a pluggable tool for your repository's performance tests. Typically, people want to run them periodically over various datasets and/or with a number of models to catch regressions with respect to run time or memory consumption. This should be as easy as
import pytest
from time import time
from tempeh.configurations import datasets, models
@pytest.mark.parametrize('Dataset', datasets.values())
@pytest.mark.parametrize('Model', models.values())
def test_fit_predict_regression(Dataset, Model):
dataset = Dataset()
X_train, X_test = dataset.get_X()
y_train, y_test = dataset.get_y()
model = Model()
max_execution_time = get_max_execution_time(dataset, model)
if model.compatible_with_dataset(dataset):
start_time = time()
model.fit(X_train, y_train)
model.predict(X_test)
duration = time() - start_time
assert duration < max_execution_time
Installation
tempeh depends on various packages to provide models, including tensorflow
, torch
, xgboost
, lightgbm
. To install a release version of tempeh
just run
pip install tempeh
Common issues
- If you're using a 32-bit Python version you might need to switch to a 64-bit Python version first to successfully install tensorflow.
- If the installation of
torch
fails try using the recommendation from the pytorch website for stable versions without CUDA for your python version on your operating system. - If the installation of
lightgbm
orxgboost
fails try to use a pip version less than 20.0 until their bug is resolved.
Structure
Datasets
Datasets (located in the datasets/
directory) encapsulate different datasets used for testing.
To add a new one
- Create a python file in the
datasets/
directory with naming convention[name]_datasets.py
- Subclass
BasePerformanceDatasetWrapper
. The naming convention is[dataset_name]PerformanceDatasetWrapper
- In
__init__
load the dataset and callsuper().__init__(data, targets, size)
- Add the class to
__init__.py
- Make sure the class contains class variables
task
,data_type
,size
- Add an entry to the
datasets
dictionary inconfigurations.py
.
Models
Models (models/
directory) wrap different machine learning models.
To add a new one
- Create a python file in the
models/
directory with naming convention[name]_model.py
- Subclass
BaseModelWrapper
and name the class[name]ModelWrapper
- In
__init__
train the model; we expect format__init__(self, ...)
- Models must contain
tasks
andalgorithm
- Add an entry to the
models
dictionary inconfigurations.py
.
Maintainers
In alphabetical order:
Contributing
To contribute please check our Contributing Guide.
Issues
Regular (non-Security) Issues
Please submit a report through Github issues. A maintainer will respond within a reasonable period of time to handle the issue as follows:
- bug: triage as
bug
and provide estimated timeline based on severity - feature request: triage as
feature request
and provide estimated timeline - question or discussion: triage as
question
and respond or notify/identify a suitable expert to respond
Maintainers are supposed to link duplicate issues when possible.
Reporting Security Issues
Please take a look at our guidelines for reporting security issues.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file tempeh-0.1.12.tar.gz
.
File metadata
- Download URL: tempeh-0.1.12.tar.gz
- Upload date:
- Size: 22.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9565fbeb2bef5b23f3aa5412fa33956b7df520d294e27d403f1e3a02e3a26b65 |
|
MD5 | 85629780899e8039343572640a29bd60 |
|
BLAKE2b-256 | 761b6cec6ab29489a619141fa0312da7157acebdf2cf9ffe37ecfd4a61365ea1 |
File details
Details for the file tempeh-0.1.12-py3-none-any.whl
.
File metadata
- Download URL: tempeh-0.1.12-py3-none-any.whl
- Upload date:
- Size: 39.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8bf4d276d09a301ff8c5e105855d03a7d1e89f495b5ad913edaf3d5d16a39e03 |
|
MD5 | 352136e7298e661be498e64de1b7d9d8 |
|
BLAKE2b-256 | b71bd016f8d2ac250b7df50ffb630c3f71371d35ac92deb80653b0e4d2f9c1a9 |