pysio-hermes 0.2.0

A realtime physiological sensing and edge AI processing framework

Heterogeneous Edge Realtime Measurement and Execution System

A Unified Open-Source Framework for Realtime Multimodal Physiological Sensing, Edge AI, and Intervention in Closed-Loop Smart Healthcare Applications

Quickstart • Docs • GUI • Showcase • Cite • Contact

HERMES for the Greek mythology analogy of the god of communication and speed, protector of information, the gods' herald. He embodies the nature of smooth and reliable communication. His role accurately resonates with the vision of this framework: facilitate reliable and fast exchange of continuously generated multimodal physiological and external data across distributed wireless and wired multi-sensor hosts for synchronized realtime data collection, in-the-loop AI stream processing, and analysis, in intelligent med- and health-tech (wearable) applications.

HERMES offers out-of-the-box streaming integrations to a number of commercial sensor devices and systems, high resolution cameras, templates for extension with custom user devices, and a ready-made wrapper for easy PyTorch AI model insertion. It reliably and synchronously captures heterogeneous data across distributed interconnected devices on a local network in a continuous manner, and enables realtime AI processing at the edge toward personalized intelligent closed-loop interventions of the user. All continuously acquired data is periodically flushed to disk for as long as the system has disk space, as MKV/MP4 and HDF5 files, for video and sensor data, respectively.

Quickstart

Core

Create a Python 3 virtual environment python -m venv .venv (python >= 3.7).

Activate it with .venv/bin/activate for Linux or .venv\Scripts\activate for Windows.

Single-command install HERMES into your project along other dependendices.

pip install pysio-hermes

Extra

All the integrated, validated and supported sensor devices are separately installable as pysio-hermes-<subpackage_name>, like:

pip install pysio-hermes-torch

Will install the AI processing subpackage to wrap user-specified PyTorch models.

List of supported devices (continuously updated)

Some subpackages require OEM software installation, check each below for detailed prerequisites.

• torch Wrapper for PyTorch AI models
• pupillabs Pupil Labs Core smartglasses
• basler Basler cameras
• dots Movella DOTs IMUs
• mvn Xsens MVN Analyze MoCap suit
• awinda Xsens Awinda IMUs
• cometa Cometa WavePlus sEMG
• moticon Moticon OpenGo pressure insoles
• tmsi TMSi SAGA physiological signals
• vicon Vicon Nexus capture system
• moxy Moxy muscle oxygenation monitor

The following subpackages are in development.

• gui Live monitoring dashboard
• mic PC-attached generic microphone

FFmpeg (Optional)

If dealing with video or audio, you will have to install FFmpeg.

Make a copy of the examples/video_codec_<type>.yml, that matches your video encoding hardware (AMD or Intel CPU, or an NVIDIA GPU), as examples/video_codec.yml

Windows

1. Download the full build with shared libraries from gyan.dev.
2. Unpack the archive into the desired folder, like C:\Program Files\ffmpeg.
3. Add path to the FFmpeg binaries to the Path environment variable manually, or via CMD.

   SETX PATH "%PATH%;C:\Program Files\ffmpeg\bin;C:\Program Files\ffmpeg" /M
   

4. Open a new terminal window and check that FFmpeg can be correctly found by the system where ffmpeg.

Linux

1. Install with the package manager sudo apt-get install ffmpeg.
2. Check that ffmpeg is on path which ffmpeg.

Running

The system runs based on YAML configuration files, where connection to other hosts, and local or remote Producer's, Consumer's, Pipeline's.

Benchmarking

Latency

1. Install Plotly into the current virtual environment pip install plotly.
2. On each host device, run the latency evaluation automated script under test/:

   cd test
   

   as test_latency.bat for Windows or . test_latency.sh for Linux.
3. Gather generated text files from all tested devices and place in test/data/latency/<device_name> subfolders in the following structure. The folder name will be used as the trace name of the corresponding series.

   root/
   └───test
       └───data
           └───latency
               ├───laptop
               │   ├───latency_vs_frequency.txt
               │   └───latency_vs_msgsize.txt
               ├───nuc
               ├───pi
               └───server
   

4. Visualize latencies by running plot_latency.bat for Windows or . plot_latency.sh for Linux:

Documentation

Check out the full documentation site for more usage examples, architecture overview, detailed extension guide, and FAQs.

Data Annotation

We developed PysioViz a complementary dashboard based on Dash Plotly for analysis and annotation of the collected multimodal data. We use it ourselves to generate ground truth labels for the AI training workflows. Check it out and leave feedback!

Showcase

These are some of our own projects enabled by HERMES to excite you to adopt it in your smart closed-looop healthtech usecases.

AI-enabled intent prediction for high-level locomotion mode selection in a smart leg prosthesis

Realtime automated cueing for freezing-of-gait Parkinson's patients in free-living conditions

Personalized level of assistance in prolong use rehabilitation and support exoskeletons

License

This sourcecode is licensed under the MIT license - see the LICENSE file for details.

The project's logo is distributed under the CC BY-NC-ND 4.0 license - see the LOGO-LICENSE.

Citation

When using in your project, research, or product, please cite the following and notify us so we can update the index of success stories enabled by HERMES.

Acknowledgement

This project was primarily written by Maxim Yudayev while at the Department of Electrical Engineering, KU Leuven.

This study was funded, in part, by the AidWear project funded by the Federal Public Service for Policy and Support, the AID-FOG project by the Michael J. Fox Foundation for Parkinson’s Research under Grant No.: MJFF-024628, the strategic basic research project RevalExo (S001024N) funded by the Research Foundation Flanders, and the Flemish Government under the Flanders AI Research Program (FAIR).

HERMES is a "Ship of Theseus"[^1] of ActionSense that started as a fork and became a complete architectural rewrite of the system from the ground up to bridge the fundamental gaps in the state-of-the-art, and to match our research group's needs in realtime deployments and reliable data acquisition. Although there is no part of ActionSense in HERMES, we believe that its authors deserve recognition as inspiration for our system.

Special thanks for early usage, contributions, bug reports, good times during experiments, and feature requests to Juha Carlon (KU Leuven), Vayalet Stefanova (KU Leuven), Diwas Lamsal (KU Leuven), Stefano Nuzzo (VUB), Léonore Foguenne (ULiège). And for the support to prof. Benjamin Filtjens (TU Delft) and prof. Bart Vanrumste (KU Leuven).

[^1]: The Ship of Theseus is a paradoxical thought experiment of identity and persistence from Greek mythology that questions whether a ship, all of whose original parts are replaced over time, remains the same ship.