Head tracker using webcam for auralization

Description

Head tracker via camera face tracking and communication via UDP protocol.

Built on top of the Google's MediaPipe face_mesh (python release).

Folder structure:

• /EACheadtracker: Contains the source code for the HeadTracker as published in the paper.
• /test: Presents auralization experiments in MATLAB using the HeadTracker.
• /audios: The raw files for the audio examples in the paper.
• /videos: The images related to the head movements that produced the audios in the paper.
• /presentation: PDF conference presentation of the paper.

System support

Setup python environment

• This application only requires you to run pip install mediapipe==0.8.3.1. However for the sake of good practices, we recommend you create a new python enviroment and install the required libraries with:

cd internoise2021-headtracker/src/

conda create --name headtracker python=3.8

conda activate headtracker

pip install -r requirements.txt

• MediaPipe supports Python 3.6 to 3.8.

Raspberry Pi

Install OpenCV and mediapipe from the sources bellow:

• OpenCV: https://pimylifeup.com/raspberry-pi-opencv/
• Mediapipe: https://pypi.org/project/mediapipe-rpi4/

Using the HeadTracker

The application can be initialized with the default parameters by running:

python HeadTracker.py

It is also possible to specify some other useful parameter by adding parameter/value flags during initialization, such as:

python HeadTracker.py --input_id 0 --port 5555 --width 1280 --height 720

Use python HeadTracker.py --help to see all the available options.

• Alternatively you may use the Windows executables distributed here. Notice that you don't need to setup an environment, or install anything else, in order to use the .exe standalones. (The distributed executables are outdated in relation to this repository)

• Connect to any plataform that accepts UDP/IP connection, use the address: IP:'127.0.0.1' and PORT:5555 .

• In order to close the app, mouse clicking "quit the window" might not work in all operating systems, as a general rule use "Esc" to finish the process.

Interpreting received data

The HeadTracker application currently sends to the server yaw, pitch and roll information in degrees and translational positions in centimeters, where downwards pitch and counterclockwise roll and yaw are denoted with negative angles, such that the full rotation is bounded between -180° and 180°, as illustrated bellow.

The sent data are strings encoded into bytes, for e.g. if the sent/received message is: b'-5,10,0,30,9,75', the corresponding coordinates are yaw=-5°, pitch=10°, roll=0°, Tx=30 cm, Ty=9 cm and Tz=75 cm — depending on the application the data needs to be decoded for proper use.

Example: reading HeadTracker output data with Matlab

Bellow you can find a snippet of how to connect to the UDP address and convert the binary data to matlab array.

% Open the HeadTracker application (make sure the file path is added to matlab path variables)
open('HeadTracker.exe')

% Connect to the local server
udpr = dsp.UDPReceiver('RemoteIPAddress', '127.0.0.1',...
                       'LocalIPPort',5555);

% Read data from the head tracker
while true
    py_output = step(udpr);
    if ~isempty(py_output)
        data = str2double(split(convertCharsToStrings(char(py_output)), ','));
        disp([' yaw:', num2str(data(1)),...
             ' pitch:', num2str(data(2)),...
             ' roll:', num2str(data(3))])
    end
end

Other examples of the connection to matlab are posted here.

Cite us

D. R. Carvalho; W. D’A. Fonseca; J. Hollebon; P. H. Mareze; F. M. Fazi. Head tracker using webcam for auralization. In 50th International Congress and Exposition on Noise Control Engineering — Internoise 2021, pages 5071–5082(12), Washington, DC, USA, Aug. 2021. doi: 10.3397/IN-2021-2956.

Bibtex:

@InProceedings{headtracker:2021,
  author    = {Davi Rocha Carvalho and William {\relax D'A}ndrea Fonseca and Jacob Hollebon and Paulo Henrique Mareze and Filippo Maria Fazi},
  booktitle = {{50th International Congress and Exposition on Noise Control Engineering --- Internoise 2021}},
  title     = {Head tracker using webcam for auralization},
  year      = {2021},
  address   = {Washington, DC, USA},
  month     = {Aug.},
  pages     = {5071--5082(12)},
  doi       = {10.3397/IN-2021-2956},
}