nosedrive 0.0.1

Simulate gamepad or mouse/keyboard inputs using Mediapipe tracking of Face joints.

Nose Drive

Simulate Xbox360 controller or mouse/keyboard inputs using Mediapipe tracking of Face joints.

This project is realized by the non-profit "ASPHI Onlus" and its main purpose is to help people with complex motor disabilities play videogames through alternative and highly-customizable inputs solutions.

Nose Drive provides a Python library that can be used to easily bind face movements to gamepad/keyboard/mouse buttons pressed by a virtual device.

AUTHORS: ASPHI Onlus

Demo video

EASY Setup Instructions

• Install the ViGEmBus driver
• Download and extract our latest release ZIP file
• Double click on the "START.bat" file
• Wait for your webcam image to show up
• Stand still in front of the webcam in your base position for 5 seconds (can be edited)
• Start playing your games
• The "N" key on the keyboard can always be pressed to reset the base position to the actual position
• The "P" key pauses and resumes inputs simulation
• If the program does not run, try to install the Visual C++ redistributable package

Inputs

Core commands:

• NOSE DOWN: gamepad analog Right-Trigger (accelerator)
• NOSE UP: gamepad Left-Trigger (brake)
• NOSE RIGHT and LEFT: gamepad Left Analog Horizontal Axis (steer left and right)

Advanced customization

Using the Nose Drive Python library it is possible to create custom scripts to customize control of different kind of videogames. Examples of such scripts can be found inside the test folder.

In the drive_settings.json file you can modify:

• sensibility values for inputs
• how many seconds to wait, after the webcam image shows up, before the program takes your actual position as the base position (the pressure of the keys is based on the deviation from this base position)
• the index of the camera that should be used for tracking, if more than one cameras are connected (default is 0)
• the keyboard keys to trigger reset of base position or to pause the program

Mediapipe Documentation

Python API examples

Videogames Controller

Driver

Driver to Emulate Xbox360 Controller

Python Libraries

• Simulate Xbox360 Controller inputs: vgamepad
• Read keyboard inputs: keyboard
• Simulate keyboard inputs: pynput

Working with Conda and Python

• Install Miniconda
• (Optional) You can add Anaconda prompt to right click menu following these steps
• Open Anaconda prompt and navigate to Nose Drive root folder
• conda create -n py310 python=3.10
• conda activate py310
• pip install -r requirements.txt
• Then you can execute every script with python, for example the command "python Drive.py drive_settings.json" will execute the Drive.py script with the settings defined in the file drive_settings.json

Build instructions

• .\python_310\Scripts\pyinstaller.exe .\Builder.spec
• Change line 6 of Builder.spec to match the .py script you want to build
• Execute compiled binaries with ".\dist\main\main.exe .\your_settings.json"

Integrations

Voice Controls

VOICE ATTACK software provides voice-activated control to simulate mouse/keyboard inputs and much more! This feature complements the nose drive very effectively, allowing to play many games that require a large number of inputs.

Eye-Tracking

Thanks to BEAM EYE TRACKER we are now able to simulate gamepad and mouse/keyboard inputs by moving our eyes!

• The scripts in this folder make use of this amazing feature.
• Of course, it is possible to combine mediapipe tracking of facial joints with the Eye gaze tracking to achieve even more customization! Note that in order to execute this kind of scripts (for example Eye_3rd_Gamepad.py) a software like "Split Cam" is required, because the webcam image should go to both Beam Eyetracker software and Python CV2 library at the same time.

License

Exept where otherwise stated, the files in this repository were created by Fondazione ASPHI Onlus and are licensed under a GNU GENERAL PUBLIC LICENSE.

• Files inside eyeware folder are intellectual property of "Beam Eye Tracker", and are thus licensed under their terms (Eyeware Tech SA. All rights reserved).
• Files inside VoiceAttack_Profiles folder were generated by the software VoiceAttack

Credits

A big "Thank you!" goes to

• the authors of ViGEmBus driver
• the authors of Mediapipe high-fidelity body pose tracking solution
• the authors of vgamepad python library
• the authors of keyboard python library
• the authors of pynput python library

on whose work our solution relies