HERMES - PyTorch

Support package to inject PyTorch AI models in the sensing-processing closed-loop of HERMES.

[!NOTE] In principle, any PyTorch model that runs in your existing AI workflow (e.g. on your laptop, single-board computer, server, edge device, etc.) is runnable via this wrapper.

• Classifier model wrapper, via TorchClassifierPipeline.
• Regression model wrapper.
• ...

Installation

Nodes available under the same HERMES namespace of hermes.torch, such as TorchClassifierPipeline.

[!NOTE] On Windows, sometimes Visual Studio Redistributable and Runtime are not installed, which would throw dynamic library load errors when importing torch. Make sure to download it if you have a clean system and are running into issues importing PyTorch on your system.

From PyPI

pip install pysio-hermes-torch

From source

git clone https://github.com/maximyudayev/hermes-torch.git
pip install -e hermes-torch

Usage

Using PyTorch AI models follows the standard configuration file specification process of HERMES nodes.

1. Prepare your PyTorch model in a regular workflow - design, train, and export a .pth checkpoint.
2. Provide the path in the HERMES config file to both, the checkpoint file and the module containing the nn.Module architecture.
3. Specify the input modalities the AI model should receive for its computations under stream_in_specs.
4. Override forward and (optionally) load_state_dict methods on your custom nn.Module class, where needed. Realtime TCN example implementation for the freezing-of-gait proof-of-concept uses internal logic for non-torch filtering and normalization, and contains non-torch object parameters: it required a method override for successful loading of the model's state dictionary from a trained checkpoint file.

[!IMPORTANT] Ensure that the state dictionary of the .pth can be successfully loaded into the model: provide the same model hyperparameters as the ones used to construct the model for training under stream_out_spec.module_params.

[!IMPORTANT] HERMES will push data from the middleware to the model input in process_data in an event-driven way. Samples from one or more modalities may be available in each iteration of the call. It is currently the responsibility of the user to buffer or fuse the incoming multi-modal samples within the nn.Module to build up receptive fields of desired function.

(Optional) Dedicated Node

You can optionally inherit from TorchClassifierPipeline, or Pipeline directly, to create a dedicated (hardcoded) Node for the target AI algorithm, similar to the HERMES extension process with support for new sensors (e.g. a TorchRegressionPipeline, TorchEmbeddingPipeline, etc., some of which are on the current development roadmap). This will simplify the YAML file structure, especially for big models. In that case, make sure to override the required abstract methods to enable auto-discovery and connection of the custom node to the rest of HERMES.

Citation

When using any parts of this repository outside of its intended use, please cite the parent project HERMES.