sotabencheval is a framework-agnostic library that contains a collection of deep learning benchmarks you can use to benchmark your models. It can be used in conjunction with the sotabench service to record results for models, so the community can compare model performance on different tasks, as well as a continuous integration style service for your repository to benchmark your models on each commit.
Benchmarks Supported
- ADE20K (Semantic Segmentation)
- COCO (Object Detection)
- ImageNet (Image Classification)
- SQuAD (Question Answering)
- WikiText-103 (Language Modelling)
- WMT (Machine Translation)
PRs welcome for further benchmarks!
Installation
Requires Python 3.6+.
pip install sotabencheval
Get Benching! 🏋️
You should read the full documentation here, which contains guidance on getting started and connecting to sotabench.
Integration is lightweight. For example, if you are evaluating an ImageNet model, you initialize an Evaluator object and (optionally) link to any linked paper:
from sotabencheval.image_classification import ImageNetEvaluator
evaluator = ImageNetEvaluator(
model_name='FixResNeXt-101 32x48d',
paper_arxiv_id='1906.06423')
Then for each batch of predictions your model makes on ImageNet, pass a dictionary of keys as image IDs and values as a np.ndarrays of logits to the evaluator.add method:
evaluator.add(output_dict=dict(zip(image_ids, batch_output)))
The evaluation logic just needs to be written in a sotabench.py file and sotabench will run it on each commit and record the results:
Contributing
All contributions welcome!
3 maintainers
Metadata
3 maintainers
Release files for sotabencheval 0.0.38
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| sotabencheval-0.0.38.tar.gz | 35.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| sotabencheval-0.0.38-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 91.3 kB
Release files / sotabencheval-0.0.38.tar.gz
| Download URL | sotabencheval-0.0.38.tar.gz |
|---|---|
| Size | 35.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
64d6b93ffb72feaef301748c2b35d987b23c7242ee96b9d38ec270447b3f789e
|
|
BLAKE2b-256 checksum How to use checksums |
d169c18b2b925f8e1d8b1dfd9bac1c3338a7e06f2054003a470f1b161f5d4bb0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/44.0.0.post20200106 requests-toolbelt/0.9.1 tqdm/4.40.0 CPython/3.7.6
|
Release files / sotabencheval-0.0.38-py3-none-any.whl
| Download URL | sotabencheval-0.0.38-py3-none-any.whl |
|---|---|
| Size | 55.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3608f722646b46663e01fdb1189d61e84ddcf6e60d10837b7b21dda8e2cea04a
|
|
BLAKE2b-256 checksum How to use checksums |
cf9dde6c4f490932b4018a04897011e34e99a6cc000f81c9fc202933bdec6448
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.2.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/44.0.0.post20200106 requests-toolbelt/0.9.1 tqdm/4.40.0 CPython/3.7.6
|
