mp2signal 0.0.2

Normalizes the mediapipe pose in 3d space based on body proportions and returns joint angles.

Mediapipe to Signal

module for processing mediapipe's holistic model for sign languages.

The module allows extracting pure movement features by:

• It provides pose normalization in 3D space using body proportions
• It rotates the pose toward the camera
• Sign movement is presented in a series of signals (directional cosine angles per each joint)
• Sign movement is easily stored, modified, processed, and generated
• Sign movement can be processed as a signal or as an image
• It has a "human-readable" representation for quick analysis and intuition.
• Simplified data representation speeds up processing and reduces storage.

Converting videos to a series of signals that are easy to read, store, and generate:

Install

pip install mp2signal

Usage

importing the module:

import mp2signal.mp2s as mp2s

Using MediaPipe we process the video and save the normalized signal:

m = mp2s.Movement('path_to_your_signlanguage_video.mp4')

The joint signals can be plotted:

import matplotlib.pyplot as plt

human_redable_posegram = m.posegram(human=True)
print(human_redable_posegram.dtype, human_redable_posegram.shape)
plt.imshow(human_redable_posegram)
plt.show()

for processing, a two-dimensional version is available in uint8 and float:

machine_uint8_posegram = m.posegram(human=False,signal=False)
print(machine_uint8_posegram.dtype, machine_uint8_posegram.shape)
plt.imshow(machine_uint8_posegram,cmap='brg')
plt.show()

machine_float_posegram = m.posegram(human=False,signal=True)
print(machine_float_posegram.dtype, machine_float_posegram.shape)
plt.imshow(machine_float_posegram,cmap='brg')
plt.show()

Storing or Generating Movement Data

Sign Language movement is easy to store and generate in the signal form.

Movement() can be initialized with the stored or generated sample:

import numpy as np
np.save('mov_human.npy',human_redable_posegram)
np.save('mov_uin8.npy',machine_uint8_posegram)
np.save('mov_float.npy',machine_float_posegram)

human_redable = np.load('mov_human.npy')
machine_uint8 = np.load('mov_uin8.npy')
machine_float = np.load('mov_float.npy')

hum_restore = mp2s.Movement(human_redable)
hum_rest_gram = hum_restore.posegram(human=True)
print(hum_rest_gram.dtype)
plt.imshow(hum_rest_gram,cmap='brg')
plt.show()

uin8_restore = mp2s.Movement(machine_uint8)
uin8_rest_gram = uin8_restore.posegram(human=False,signal=False)
print(uin8_rest_gram.dtype)
plt.imshow(uin8_rest_gram,cmap='brg')
plt.show()

float_restore_float = mp2s.Movement(machine_float)
float_rest_gram = float_restore_float.posegram(human=False,signal=True)
print(float_rest_gram.dtype)
plt.imshow(float_rest_gram,cmap='brg')
plt.show()

Signals to Video

You can generate a video with your stored/generated movement signals:

video_out = uin8_restore.make_a_video('video_from_signal.mp4')

Usage with Camera

This script takes input from the web camera and shows rotated and normalized skeleton:

python examples/mp_live_rot_demo.py