neurobench 2.3.0

NeuroBench Structure

NeuroBench contains the following sections:

Installation

Install from PyPI:

pip install neurobench

Benchmarks

The following benchmarks are currently available:

Getting started

Example benchmark scripts can be found under the examples folder. (https://github.com/NeuroBench/neurobench/tree/main/examples/)

In general, the design flow for using the framework is as follows:

1. Train a network using the train split from a particular dataset.

2. Wrap the network in a NeuroBenchModel.

3. Pass the model, evaluation split dataloader, pre-/post-processors, and a list of metrics to the Benchmark and run().

Documentation for the framework interfaces can found in the API Overview.

Development

If you clone the repo directly for development, poetry can be used to maintain a virtualenv consistent with a deployment environment. In the root directory run:

pip install poetry
poetry install

Poetry requires python >=3.9. Installation should not take more than a few minutes.

End-to-end examples can be run from the poetry environment. As a demo, try the Google Speech Commands keyword classification benchmark:

# ANN Benchmark Example
poetry run python examples/gsc/benchmark_ann.py

# Expected results:
# {'Footprint': 109228, 'ConnectionSparsity': 0.0,
# 'ClassificationAccuracy': 0.8653339412687909, 'ActivationSparsity': 0.3854464619019532,
# 'SynapticOperations': {'Effective_MACs': 1728071.1701953658, 'Effective_ACs': 0.0, 'Dense': 1880256.0}}


# SNN Benchmark Example
poetry run python examples/gsc/benchmark_snn.py

# Expected results:
# {'Footprint': 583900, 'ConnectionSparsity': 0.0,
# 'ClassificationAccuracy': 0.85633802969095, 'ActivationSparsity': 0.9668664144456199,
# 'SynapticOperations': {'Effective_MACs': 0.0, 'Effective_ACs': 3289834.3206724217, 'Dense': 29030400.0}}

These demos should download the dataset, then run in a couple minutes. Other baseline result scripts and notebook tutorials are available in the examples folder.

Developers

NeuroBench is a collaboration between industry and academic engineers and researchers. This framework is currently maintained by Jason Yik, Noah Pacik-Nelson, Korneel Van den Berghe and Benedetto Leto and there have been technical contributions from many others. A non-exhaustive list includes Gregor Lenz, Denis Kleyko, Younes Bouhadjar, Paul Hueber, Vincent Sun, Biyan Zhou, George Vathakkattil Joseph, Douwe den Blanken, Maxime Fabre, Shenqi Wang, Guangzhi Tang, Anurag Kumar Mishra, Soikat Hasan Ahmed, Aurora Micheli, Tao Sun.

Contributing

If you are interested in helping to build this framework, please see the Contribution Guidelines.

Citation

If you use this framework in your research, please cite the following article:

@article{yik2025neurobench,
   title={The neurobench framework for benchmarking neuromorphic computing algorithms and systems},
   author={Yik, Jason and Van den Berghe, Korneel and den Blanken, Douwe and Bouhadjar, Younes and Fabre, Maxime and Hueber, Paul and Ke, Weijie and Khoei, Mina A and Kleyko, Denis and Pacik-Nelson, Noah and others},
   journal={Nature communications},
   volume={16},
   number={1},
   pages={1545},
   year={2025},
   publisher={Nature Publishing Group UK London}
}