neurobench 2.3.0

Collaborative, Fair, and Representative Benchmarks for Neuromorphic Computing

Introduction

A harness for running evaluations on NeuroBench algorithm benchmarks.

NeuroBench is a community-driven project, and we welcome further development from the community. If you are interested in developing extensions to features, programming frameworks, or metrics and tasks, please see the Contributing Guidelines.

NeuroBench Structure

NeuroBench contains the following sections:

Section	Description
neurobench.benchmarks	Neurobench benchmarks, including workload metrics and static metrics
neurobench.datasets	Neurobench benchmark datasets
neurobench.models	Neurobench framework for Torch and SNNTorch models
neurobench.preprocessing	Pre-processing of data, conversion to spikes
neurobench.postprocessing	Post-processors take the spiking output from the models and provide several methods of combining them

Installation

Install from PyPI:

pip install neurobench

Benchmarks

The following benchmarks are currently available:

v1.0 benchmarks

• Keyword Few-shot Class-incremental Learning (FSCIL)

• Event Camera Object Detection

• Non-human Primate (NHP) Motor Prediction

• Chaotic Function Prediction

Leaderboards

Proposed solutions for the benchmark tasks are evaluated on a set of metrics and compared to the performance of other solutions.

The leaderboards for these benchmarks can be found here.

Additional benchmarks

• DVS Gesture Recognition

• Google Speech Commands (GSC) Classification

• Neuromorphic Human Activity Recognition (HAR)

Getting started

Example benchmark scripts can be found under the examples folder. (https://github.com/NeuroBench/neurobench/tree/main/examples/)

In general, the design flow for using the framework is as follows:

1. Train a network using the train split from a particular dataset.

2. Wrap the network in a NeuroBenchModel.

3. Pass the model, evaluation split dataloader, pre-/post-processors, and a list of metrics to the Benchmark and run().

Documentation for the framework interfaces can found in the API Overview.

Development

If you clone the repo directly for development, poetry can be used to maintain a virtualenv consistent with a deployment environment. In the root directory run:

pip install poetry
poetry install

Poetry requires python >=3.9. Installation should not take more than a few minutes.

End-to-end examples can be run from the poetry environment. As a demo, try the Google Speech Commands keyword classification benchmark:

# ANN Benchmark Example
poetry run python examples/gsc/benchmark_ann.py

# Expected results:
# {'Footprint': 109228, 'ConnectionSparsity': 0.0,
# 'ClassificationAccuracy': 0.8653339412687909, 'ActivationSparsity': 0.3854464619019532,
# 'SynapticOperations': {'Effective_MACs': 1728071.1701953658, 'Effective_ACs': 0.0, 'Dense': 1880256.0}}


# SNN Benchmark Example
poetry run python examples/gsc/benchmark_snn.py

# Expected results:
# {'Footprint': 583900, 'ConnectionSparsity': 0.0,
# 'ClassificationAccuracy': 0.85633802969095, 'ActivationSparsity': 0.9668664144456199,
# 'SynapticOperations': {'Effective_MACs': 0.0, 'Effective_ACs': 3289834.3206724217, 'Dense': 29030400.0}}

These demos should download the dataset, then run in a couple minutes. Other baseline result scripts and notebook tutorials are available in the examples folder.

Developers

NeuroBench is a collaboration between industry and academic engineers and researchers. This framework is currently maintained by Jason Yik, Noah Pacik-Nelson, Korneel Van den Berghe and Benedetto Leto and there have been technical contributions from many others. A non-exhaustive list includes Gregor Lenz, Denis Kleyko, Younes Bouhadjar, Paul Hueber, Vincent Sun, Biyan Zhou, George Vathakkattil Joseph, Douwe den Blanken, Maxime Fabre, Shenqi Wang, Guangzhi Tang, Anurag Kumar Mishra, Soikat Hasan Ahmed, Aurora Micheli, Tao Sun.

Contributing

If you are interested in helping to build this framework, please see the Contribution Guidelines.

Citation

If you use this framework in your research, please cite the following article:

@article{yik2025neurobench,
   title={The neurobench framework for benchmarking neuromorphic computing algorithms and systems},
   author={Yik, Jason and Van den Berghe, Korneel and den Blanken, Douwe and Bouhadjar, Younes and Fabre, Maxime and Hueber, Paul and Ke, Weijie and Khoei, Mina A and Kleyko, Denis and Pacik-Nelson, Noah and others},
   journal={Nature communications},
   volume={16},
   number={1},
   pages={1545},
   year={2025},
   publisher={Nature Publishing Group UK London}
}