Skip to main content

Easily benchmark Machine Learning models on selected tasks and datasets

Project description

PyPI version Generic badge

sotabencheval is a framework-agnostic library that contains a collection of deep learning benchmarks you can use to benchmark your models. It can be used in conjunction with the sotabench service to record results for models, so the community can compare model performance on different tasks, as well as a continuous integration style service for your repository to benchmark your models on each commit.

Benchmarks Supported

PRs welcome for further benchmarks!


Requires Python 3.6+.

pip install sotabencheval

Get Benching! 🏋️

You should read the full documentation here, which contains guidance on getting started and connecting to sotabench.

Integration is lightweight. For example, if you are evaluating an ImageNet model, you initialize an Evaluator object and (optionally) link to any linked paper:

from sotabencheval.image_classification import ImageNetEvaluator
evaluator = ImageNetEvaluator(
             model_name='FixResNeXt-101 32x48d',

Then for each batch of predictions your model makes on ImageNet, pass a dictionary of keys as image IDs and values as a np.ndarrays of logits to the evaluator.add method:

evaluator.add(output_dict=dict(zip(image_ids, batch_output)))

The evaluation logic just needs to be written in a file and sotabench will run it on each commit and record the results:


All contributions welcome!

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for sotabencheval, version 0.0.38
Filename, size File type Python version Upload date Hashes
Filename, size sotabencheval-0.0.38-py3-none-any.whl (55.4 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size sotabencheval-0.0.38.tar.gz (35.9 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page