nodify-plugin 0.1.2

Nodify Tensorboard Plugin.

Nodify Tensorboard Plugin

This is a tensorboard plugin by Trainy to supplement the existing PyTorch Profiler. This provides additional visualizations to effectively characterize traces for runs involving multiple GPUs. The plugin expects all traces to be collected using torch.profile and to be located in the same folder.

Installation & Quickstart

Install tensorboard and the plugin.

pip install tensorboard
pip install nodify-plugin

Generate PyTorch profiler traces as shown here and bring up the tensorboard where your traces are living. A set of example logs are provided in this repo under log/resnet18

tensorboard --logdir log/resnet18/

Take a look at our quickstart guide to learn about the different graph views and how you can use them to debug your multinode training.

Development

To view the plugin for development, create a virtual environment, install the requirements, and install the plugin.

python -m venv venv
. venv/bin/activate
pip install -e .

Feature roadmap

A lot of the features on the roadmap use Meta's Dynolog, kineto, and holistic trace analyzer.

• On-demand tracing and metrics through dynologger
• Recommendations for fixing multinode bottlenecks
• reading from logs stored on cloud object-stores (e.g. Amazon S3, Azure Blob)

Contributing

For feature requests or bug reports/fixes, feel free to open a Github issue or make a pull request. We'd love to connect with any interested developers and we check our Discord to discuss the direction of our projects everyday. Connect with us either throught the text channels or through DM's.