Goal

There are various handshape datasets for Sign Language. However:

• Each dataset has its own format and many are hard to find.
• Each dataset has its own mapping of handshapes to classes. While signs depend on the specific Sign Language for a country/region, handshapes are universal. Hence, they could be shared between datasets/tasks.

This library aims to provide two main features:

• A simplified API to download and load handshape datasets
• A mapping between datasets so that datasets can be merged for training/testing models.

This library is a work in progress. Contributions are welcome. If you wish to add a dataset you can make a push request or open an issue.

Installation

You can install handshape_datasets via pip with:

pip install handshape_datasets

Basic usage

Simply import the module and load a dataset. The following downloads, preprocesses and load to memory the LSA16 dataset:

import handshape_datasets as hd
images,metadata = hd.load("lsa16")

Afterwards you can, for example, plot the first images of the dataset

import matplotlib.pyplot as plt
plt.imshow(images[0,:,:,:]) # show the first sample of the dataset

Advanced usage

import handshape_datasets as hd
hd.list_datasets() # List available datasets
hd.load("lsa16",version="color",delete=True) # use the color version, delete temporary files
hd.delete_temporary_files("lsa16")# Delete temporary files  (if any)
hd.clear("lsa16") # Delete all the local files for dataset LSA16
hd.info("lsa16") # Shows detailed info of the dataset, including url, data format, fields, etc.

Supported datasets

You can find more information about the datasets in the following sign language dataset survey

Training a handshape classifier with Keras

Load the dataset:

x,metadata = handshape_datasets.load("lsa16")
y = metadata["y"]

Get the input_shape and number of classes:

input_shape = x[0].shape
classes = y.max() + 1

Define a model (using a pretrained MobileNet here):

base_model = keras.applications.mobilenet.MobileNet(input_shape=(input_shape[0],input_shape[1],3), 
                                                            weights='imagenet', include_top=False)
output = keras.layers.GlobalAveragePooling2D()(base_model.output)
output = keras.layers.Dense(32, activation='relu')(output)
output = keras.layers.Dense(classes, activation='softmax')(output)
model = Model(inputs=base_model.input, outputs=output)
model.compile(optimizer='Adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

Split the dataset intro train/test sets:

X_train, X_test, Y_train, Y_test = sklearn.model_selection.train_test_split(x,y,
                                                                                test_size=0.9,
                                                                                stratify=y)

Fit the model

history = model.fit(X_train, Y_train, batch_size=self.batch_size, epochs=self.epochs, validation_data=(X_test, Y_test))

Google Colab example:

https://colab.research.google.com/drive/1kY-YrbegGFVT7NqVaeA4RjXYRVlZiISR?usp=sharing