nnsmith 0.0.1

"Automatic DNN generation for fuzzing and more."

Backend-Model Support

Model\Engine	TVM	ORT	TensorRT	TFLite	XLA	IREE
PyTorch-ONNX	✅	✅	✅
TensorFlow				⚠️	⚠️	⚠️

✅: Supported; ⚠️: Beta support; Others are not supported yet -- Contributions are welcome!

Setup

Install latest stable release:

pip install "nnsmith[torch,onnx]" --upgrade

Install GitHub HEAD: [click to expand]

pip install "git+https://github.com/ise-uiuc/nnsmith@main#egg=nnsmith[torch,onnx]" --upgrade
# or pip install "git+ssh://[e-mail]/ise-uiuc/nnsmith@main#egg=nnsmith[torch,onnx]" --upgrade

Install latest pre-release: [click to expand]

pip install "nnsmith[torch,onnx]"                     \
            --pre --upgrade                           \
            --index-url https://test.pypi.org/simple/ \
            --extra-index-url https://pypi.org/simple/

Quick Start

Setting up graphviz for debugging [click to expand]

Graphviz provides dot for visualizing graphs in nice pictures. But it needs to be installed via the following methods:

sudo apt-get install graphviz graphviz-dev      # Linux
brew install graphviz                           # MacOS
conda install --channel conda-forge pygraphviz  # Conda
choco install graphviz                          # Windows

pip install pygraphviz  # Final step.

Also see pygraphviz install guidance.

# Generate a random model in "nnsmith_outputs/*"
nnsmith.model_gen model.type=onnx debug.viz=true

See other commands under doc/cli. We use hydra to manage configurations. See nnsmith/config/main.yaml.

Developer Notes

• pip install -r requirements/core.txt to run generation and fuzzing;
• pip install --upgrade -r requirements/sys/[system].txt to allow generating and running specific frameworks;
  • Why "--upgrade"? In fact, all the sources under requirements/sys/ are nightly release (except tvm) as we want to "save the world" by catching new bugs;

Pre-commits [click to expand]

You can use pre-commit to simpify development:

• pip install -r requirements/dev.txt;
• pre-commit install;
• pre-commit will run upon a commit; To explicitly run pre-commit for all files: pre-commit run --all-files.

Local development [click to expand]

• Develop locally by setting export PYTHONPATH=$PYTHONPATH:$(pwd) (pwd should be this git folder.)
• Set PYTHONPATH="" when doing pip install nnsmith from online version.

Simplify the code [click to expand]

Simplicity is prerequisite for reliability. --Edsger W. Dijkstra

We want code simplicity: keeping minimal dependencies and focusing on a small set of simple APIs to make NNSmith maintainable to developers and reliable to users.

Test before commit [click to expand]

# env of torch & tf will conflict so split their unit tests.
pytest tests/core -s
pytest tests/torch -s
pytest tests/tensorflow -s

Paper

Our paper is accepted by ASPLOS'23 and the pre-print is now available on .