Skip to main content

Navigate Desktop with Gestures

Project description

Gestop : Customizable Gesture Control of Computer Systems

This is the implementation of the approach described in the paper:

Sriram Krishna and Nishant Sinha. Gestop: Customizable Gesture Control of Computer Systems 8th ACM IKDD CODS and 26th COMAD. 2021. 405-409.

Built on top of mediapipe, this project aims to be a tool to interact with a computer through hand gestures. Out of the box, using this tool, it is possible to:

  1. Use your hand to act as a replacement for the mouse.
  2. Perform hand gestures to control system parameters like screen brightness, volume etc.

In addition, it is possible to extend and customize the functionality of the application in numerous ways:

  1. Remap existing hand gestures to different functions in order to better suit your needs.
  2. Create custom functionality through the use of either python functions or shell scripts.
  3. Collect data and create your own custom gestures to use with existing gestures.

Demo (Click on the image to see the full video)

Demo video link

Static and Dynamic Gestures Dataset link

Installation

Installation using pip inside a virtual environment is highly recommended. To do so:

python -m venv env
source env/bin/activate
pip install gestop

In addition to the Python dependencies, OpenCV and xdotool are also required by Gestop.

Usage

Server

To start the Gestop server:

python -m gestop.receiver

Client

The client, or the keypoint generator, can be setup either through MediaPipe's C++ API, or through its Python API. The Python API is simpler to setup and is recommended.

MediaPipe Python API

python -m gestop.keypoint_gen.hand_tracking
MediaPipe C++ API
  1. Download mediapipe and set it up. MediaPipe >=0.8.0 is NOT supported and should no be used. Make sure the provided hand tracking example is working to verify if all dependencies are installed.
  2. Clone this repo in the top level directory of mediapipe. Install all of Gestop's dependencies.
  3. Run the instructions below to build and then execute the code.

Note: Run build instructions in the mediapipe/ directory, not inside this directory.

GPU (Linux only)
bazel build -c opt --verbose_failures --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 gestop:hand_tracking_gpu

GLOG_logtostderr=1 bazel-bin/gestop/hand_tracking_gpu --calculator_graph_config_file=gestop/gestop/keypoint_gen/hand_tracking_desktop_live.pbtxt
CPU
bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 gestop:hand_tracking_cpu

GLOG_logtostderr=1 bazel-bin/gestop/hand_tracking_cpu --calculator_graph_config_file=gestop/keypoint_gen/hand_tracking_desktop_live.pbtxt

Overview

The hand keypoints are detected using google's MediaPipe. These keypoints are then fed into receiver.py . The tool recognizes two kinds of gestures:

  1. Static Gestures : Gestures whose meaning can be inferred from a single image itself.
  2. Dynamic Gestures : Gestures which can only be understood through a sequence of images i.e. a video.

Static gestures, by default, are mapped to all functionality relevant to the mouse, such as left mouse click, scroll etc. Combined with mouse tracking, this allows one to replace the mouse entirely. The mouse is tracked simply by moving the hand, where the tip of the index finger reflects the position of the cursor. The gestures related to the mouse actions are detailed below. To train the neural network to recognize static gestures, a dataset was created manually for the available gestures.

For more complicated gestures involving the movement of the hand, dynamic gestures can be used. By default, it consists of various other actions to interface with the system, such as modifying screen brightness, switching workspaces, taking screenshots etc. The data for these dynamic gestures comes from SHREC2017 dataset. Dynamic gestures are detected by holding down the Ctrl key, which freezes the cursor, performing the gesture, and then releasing the key.

The project consists of a few distinct pieces which are:

  • MediaPipe - Accessed through either the Python API or the C++ API, MediaPipe tracks the hand, generates the keypoints and transmits them.
  • Gesture Receiver - See receiver.py, responsible for handling the stream and utilizing the following modules.
  • Mouse Tracker - See mouse_tracker.py, responsible for moving the cursor using the position of the index finger.
  • Gesture Recognizer - See recognizer.py, takes in the keypoints from the mediapipe executable, and converts them into a high level description of the state of the hand, i.e. a gesture name.
  • Gesture Executor - See executor.py, uses the gesture name from the previous module, and executes an action.

Notes

  • For best performance, perform dynamic gestures with right hand only, as all data from SHREC is right hand only.
  • For dynamic gestures to work properly, you may need to change the keycodes being used in executor.py. Use the given find_keycode.py script to find the keycodes of the keys used to change screen brightness and volumee. Finally, system shortcuts may need to be remapped so that the shortcuts work even with the Ctrl key held down. For example, in addition to the usual default behaviour of <Prnt_Screen> taking a screenshot, you may need to add <Ctrl+Prnt_Screen> as a shortcut as well.

Customizing Gestop

Available Gestures

API Reference

Useful Information

Joints of the hand

HandCommander

Video recorded with VokoScreenNG

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

gestop-1.0.1.tar.gz (349.4 kB view details)

Uploaded Source

Built Distribution

gestop-1.0.1-py3-none-any.whl (351.4 kB view details)

Uploaded Python 3

File details

Details for the file gestop-1.0.1.tar.gz.

File metadata

  • Download URL: gestop-1.0.1.tar.gz
  • Upload date:
  • Size: 349.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.9

File hashes

Hashes for gestop-1.0.1.tar.gz
Algorithm Hash digest
SHA256 abb79119345833127617e16873f094f3e5b5427ee6b571d63b872d3c88b565af
MD5 3fe54d940a3093b4aa76c717caa0deb7
BLAKE2b-256 b4a4434767016ec7952895f46b32f70ed534141d012688f5913ab5f34e7004ed

See more details on using hashes here.

File details

Details for the file gestop-1.0.1-py3-none-any.whl.

File metadata

  • Download URL: gestop-1.0.1-py3-none-any.whl
  • Upload date:
  • Size: 351.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.10.9

File hashes

Hashes for gestop-1.0.1-py3-none-any.whl
Algorithm Hash digest
SHA256 fb3b3f53e706709fa062801b1f9c54c472eb449f10adb2d363bb19171de45e9b
MD5 553d38fba0394c0bd879be0de402ee93
BLAKE2b-256 7ee96a19719305a6f8e8639c7273c2d26758bd3546515f1f9983a01ccd001a1e

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page