nasbenchapi 1.0.1

Lightweight, unified pickle-based NASBench APIs (101/201/301) with downloader

NASBenchAPI

A unified, lightweight interface for NASBench-101, 201, and 301 with optimized Pickle-based datasets.

---

Getting Started

NASBenchAPI is a lightweight, unified interface for Neural Architecture Search benchmarks (101, 201, and 301). All NASBench datasets (originally in .tfrecord, .pth, and .json formats) were extracted and saved as Pickle-based files for consistency.

Related Works

This project is inspired by the holistic NAS Library, NASLib, and the paper by Mehta et al..

The primary motivation for NASBenchAPI stems from the need to integrate NASBench datasets (101, 201, 301) into custom frameworks without the significant overhead and extraneous tools introduced by more comprehensive libraries. This API provides a focused, lightweight, and unified interface specifically for that purpose.

Installation

PyPi (recommended)

The Python package is hosted on the Python Package Index (PyPI).

The latest published version of NASBenchAPI can be installed using

pip install nasbenchapi

Manual Installation

Simply clone the entire repo and extract the files in the nasbenchapi folder, then import them into your project folder.

Or use one of the shorthand methods below

GIT

• cd into your project directory

• Use sparse-checkout to pull the library files only into your project directory

git init nasbenchapi

cd nasbenchapi

git remote add -f origin https://github.com/ThunderStruct/NASBenchAPI.git
git config core.sparseCheckout true

echo "nasbenchapi/*"  >> .git/info/sparse-checkout

git pull --depth=1  origin  main

• Import the newly pulled files into your project folder

SVN

• cd into your project directory

• checkout the library files

svn checkout https://github.com/ThunderStruct/NASBenchAPI/trunk/nasbenchapi

• Import the newly checked out files into your project folder

Quick Start

Basic Usage

Loading and initializing a benchmark

from nasbenchapi import NASBench101, NASBench201, NASBench301

# Initialize with explicit path

nb101 = NASBench101('/path/to/nb101.pkl')  # Same for 201, 301

# Or use environment variables
# export NASBENC2101_PATH=/path/to/nb201.pkl

nb201 =  NASBench201()

Sample random architectures

archs = nb101.random_sample(n=5,  seed=42)    # randomly sample 5 architectures

print(f"Sampled {len(archs)} architectures")

Query performance of an architecture

arch = archs[0]

result = nb101.query(arch,  dataset='cifar10',  split='val')

print(f"Validation accuracy: {result['metric']}")
print(f"Training time: {result['cost']}")

Iterate over all architectures

for i, arch in  enumerate(nb101.iter_all()):
    if i >=  10:
        break
    print(f"Architecture {i}: {nb101.id(arch)}")
    

Dataset Management

Environment Variables (recommended)

Set environment variables to avoid passing paths explicitly and work seamlessly across different projects:

export NASBENCH101_PATH=/path/to/nb101.pkl
export NASBENCH201_PATH=/path/to/nb201.pkl
export NASBENCH301_PATH=/path/to/nb301.pkl

CLI Downloader (recommended)

Download the Pickle-based benchmark datasets through the CLI:

nasbench-download

You may optionally set the --benchmark={101|201|301} argument. Otherwise, the tool will prompt for benchmark selection interactively.

Manual Download

Alternatively, manually download the Pickle-based benchmarks through the following links:

Benchmark	Download Link
NASBench-101	Figshare Link
NASBench-201	Figshare Link
NASBench-301	Figshare Link

Documentation

Detailed examples and the full API docs are hosted on Read the Docs.

Benchmarks at a Glance

Benchmark	Datasets	Metrics	Search Space Size
NASBench-101	CIFAR-10	train/val/test accuracy, training time	423,624
NASBench-201	CIFAR-10, CIFAR-100, ImageNet16-120	train/val/test accuracy, losses	15,625
NASBench-301	CIFAR-10, CIFAR-100	surrogate val/test accuracy	~10^18 (surrogate)

Cite

If you use this library in your work, please use the following BibTeX entry:

@misc{nasbenchapi-2025, 
  title={NASBenchAPI: A unified interface for NASBench datasets}, 
  author={Shahawy, Mohamed}, 
  year={2025}, 
  publisher={GitHub}, 
  howpublished={\url{https://github.com/ThunderStruct/NASBenchAPI}} 
}

License

This project is licensed under the MIT License - see the LICENSE file for details