nnsmith 0.0.0

"Automatic DNN generation for fuzzing and more."

NNSmith: DNN Model Generation in the Wild

Backend-Model Support

Backend\Model	ONNX/PyTorch	TensorFlow
TVM	✅
ONNXRuntime	✅
TensorRT	✅
TFLite		⚠️
XLA		⚠️

✅: Supported; ⚠️: Beta support; Others are not supported yet -- Contributions are welcome!

Quick Start

Setting up graphviz for debugging [click to expand]

Graphviz provides dot for visualizing graphs in nice pictures. But it needs to be installed via the following methods:

sudo apt-get install graphviz graphviz-dev      # Linux
brew install graphviz                           # MacOS
conda install --channel conda-forge pygraphviz  # Conda
choco install graphviz                          # Windows

Also see pygraphviz install guidance.

python3 -m pip install "nnsmith[torch,onnx]"                     \
                       --upgrade                                 \
                       --index-url https://test.pypi.org/simple/ \
                       --extra-index-url https://pypi.org/simple/
# Generate a 5-node graph:
nnsmith.model_gen model.max_nodes=5 debug.viz=true
# see "nnsmith_outputs/*"

See other commands under nnsmith/cli. We use hydra to manage configurations. See nnsmith/config/main.yaml.

Developer Notes

• pip install -r requirements/core.txt to run generation and fuzzing;
• pip install --upgrade -r requirements/sys/[system].txt to allow generating and running specific frameworks;
  • Why "--upgrade"? In fact, all the sources under requirements/sys/ are nightly release (except tvm) as we want to "save the world" by catching new bugs;

export PYTHONPATH=$PYTHONPATH:$(pwd)

You can use pre-commit to simpify development:

• pip install -r requirements/dev.txt;
• pre-commit install;
• pre-commit will run upon a commit; To explicitly run pre-commit for all files: pre-commit run --all-files.

More notes [click to expand]

Simplicity is prerequisite for reliability. --Edsger W. Dijkstra

We want code simplicity: keeping minimal dependencies and focusing on a small set of simple APIs to make NNSmith maintainable to developers and reliable to users.

Run tests before commit:

# env of torch & tf will conflict so split their unit tests.
pytest tests/core -s
pytest tests/torch -s
pytest tests/tensorflow -s