glasses-detector 0.1.1

Eyeglasses and sunglasses detector (classifier and segmenter)

Glasses Detector

Eyeglasses and sunglasses classifier + glasses and their frames segmenter. This project provides a quick way to use the pre-trained models via python script or terminal. Based on selected task, an image or a directory of images will be processed and corresponding labels or masks will be generated.

Note: the project is BETA stage. Currently, only sunglasses classification models (except huge) and full glasses segmentation models (except medium) are available.

Installation

Minimum version of Python 3.10 is required. Also, you may want to install Pytorch in advance for your device to enable GPU support. Note that CUDA is backwards compatible, thus even if you have the newest version of CUDA Toolkit, Pytorch should work just fine.

To install the package, simply run:

pip install glasses-detector

You can also install it from source:

git clone https://github.com/mantasu/glasses-detector
cd glasses-detector && pip install .

If you want to train your own models on the same datasets (or your custom ones), check the GitHub repository.

Features

There are 2 kinds of classifiers and 2 kinds of segmenters (terminology is a bit off but easier to handle with unique names):

• Eyeglasses classifier - identifies only transparent glasses, i.e., prescription spectacles.
• Sunglasses classifier - identifies only occluded glasses, i.e., sunglasses.
• Full glasses segmenter - segments full glasses, i.e., their frames and actual glasses (regardless of the glasses type).
• Glasses frames segmenter - segments glasses frames (regardless of the glasses type).

Each kind has 5 different model architectures with naming conventions set from tiny to huge.

Classification

A classifier only identifies whether a corresponding category of glasses (transparent eyeglasses or occluded sunglasses) is present:

Model type
Eyeglasses classifier	wears	doesn't wear	doesn't wear
Sunglasses classifier	doesn't wear	wears	doesn't wear
Any glasses classifier	wears	wears	doesn't wear

These are the performances of eyeglasses and sunglasses models and their sizes. Note that the joint glasses classifier would have an average accuracy and a combined model size of both eyeglasses and sunglasses models.

Eyeglasses classification models (performance & weights)

Model type	BCE loss $\downarrow$	F1 score $\uparrow$	ROC-AUC score $\uparrow$	Num params $\downarrow$	Model size $\downarrow$
Eyeglasses classifier tiny	TBA	TBA	TBA	TBA	TBA
Eyeglasses classifier small	TBA	TBA	TBA	TBA	TBA
Eyeglasses classifier medium	TBA	TBA	TBA	TBA	TBA
Eyeglasses classifier large	TBA	TBA	TBA	TBA	TBA
Eyeglasses classifier huge	TBA	TBA	TBA	TBA	TBA

Sunglasses classification models (performance & weights)

Model type	BCE loss $\downarrow$	F1 score $\uparrow$	ROC-AUC score $\uparrow$	Num params $\downarrow$	Model size $\downarrow$
Sunglasses classifier tiny	0.1149	0.9137	0.9967	27.53 k	0.11 Mb
Sunglasses classifier small	0.0645	0.9434	0.9987	342.82 k	1.34 Mb
Sunglasses classifier medium	0.0491	0.9651	0.9992	1.52 M	5.84 Mb
Sunglasses classifier large	0.0532	0.9685	0.9990	4.0 M	15.45 Mb
Sunglasses classifier huge	TBA	TBA	TBA	TBA	TBA

Segmentation

A full-glasses segmenter generates masks of people wearing corresponding categories of glasses and their frames, whereas frames-only segmenter generates corresponding masks but only for glasses frames:

Model type
Full/frames eyeglasses segmenter
Full/frames sunglasses segmenter
Full/frames any glasses segmenter

There is only one model group for each full-glasses and frames-only segmentation tasks. Each group is trained for both eyeglasses and sunglasses. Although you can use it as is, it is only one part of the final full-glasses or frames-only segmentation model - the other part is a specific classifier, therefore, the accuracy and the model size would be a combination of the generic (base) segmenter and a classifier of a specific glasses category.

Full glasses segmentation models (performance & weights)

Model type	BCE loss $\downarrow$	F1 score $\uparrow$	Dice score $\uparrow$	Num params $\downarrow$	Model size $\downarrow$
Full glasses segmenter tiny	0.0580	0.9054	0.9220	926.07 k	3.54 Mb
Full glasses segmenter small	0.0603	0.8990	0.9131	3.22 M	12.37 Mb
Full glasses segmenter medium	TBA	TBA	TBA	TBA	TBA
Full glasses segmenter large	0.0515	0.9152	0.9279	32.95 M	125.89 Mb
Full glasses segmenter huge	0.0516	0.9147	0.9272	58.63 M	224.06 Mb

Glasses frames segmentation models (performance & weights)

Model type	BCE loss $\downarrow$	F1 score $\uparrow$	Dice score $\uparrow$	Num params $\downarrow$	Model size $\downarrow$
Glasses frames segmenter tiny	TBA	TBA	TBA	TBA	TBA
Glasses frames segmenter small	TBA	TBA	TBA	TBA	TBA
Glasses frames segmenter medium	TBA	TBA	TBA	TBA	TBA
Glasses frames segmenter large	TBA	TBA	TBA	TBA	TBA
Glasses frames segmenter huge	TBA	TBA	TBA	TBA	TBA

Examples

Command Line

You can run predictions via the command line. For example, classification of a single or multiple images, can be performed via

glasses-detector -i path/to/img --kind sunglasses-classifier # Prints 1 or 0
glasses-detector -i path/to/dir --kind sunglasses-classifier # Generates CSV

Running segmentation is similar, just change the kind argument:

glasses-detector -i path/to/img -k glasses-segmenter # Generates img_mask file
glasses-detector -i path/to/dir -k glasses-segmenter # Generates dir with masks

Note: you can also specify things like --output-path, --label-type, --size, --device etc. Use --glasses-detector -h for more details or check the documentation page.

Python Script

You can import the package and its models via the python script for more flexibility. Here is an example of how to classify people wearing sunglasses (will generate an output file where each line will contain the name of the image and the predicted label, e.g., some_image.jpg,1):

from glasses_detector import SunglassesClassifier

classifier = SunglassesClassifier(model_type="small", pretrained=True).eval()

classifier.predict(
    input_path="path/to/dir", 
    output_path="path/to/output.csv",
    label_type="int",
)

Using a segmenter is similar, here is an example of using a sunglasses segmentation model:

from glasses_detector import FullSunglassesSegmenter

# model_type can also be a tuple: (classifier size, base glasses segmenter size)
segmenter = FullSunglassesSegmenter(model_type="small", pretrained=True).eval()

segmenter.predict(
    input_path="path/to/dir",
    output_path="path/to/dir_masks",
    mask_type="img",
)

Note: there is much more flexibility that you can do with the given models, for instance, you can use only base segmenters without accompanying classifiers, or you can define your own prediction methods without resizing images to 256x256 (as what is done in the background). For more details refer to the documentation page, for instance at how segmenter prediction method works.

Demo

Feel free to play around with some demo image files. For example, after installing through pip, you can run:

git clone https://github.com/mantasu/glasses-detector && cd glasses-detector/data
glasses-detector -i demo -o demo_labels.csv --kind sunglasses-classifier --label str

References

The following model architectures were used from Torchvision library:

• Classifier small - ShuffleNet V2 (x0.5) based on ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design paper
• Classifier medium - MobileNet V3 (small) based on Searching for MobileNetV3 paper
• Classifier large - EfficientNet B0 based on EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks paper
• Segmenter small - LRASPP based on Searching for MobileNetV3 paper
• Segmenter medium - FCN (ResNet-50) based on Fully Convolutional Networks for Semantic Segmentation paper
• Segmenter large - DeepLab V3 (ResNet-101) based on Rethinking Atrous Convolution for Semantic Image Segmentation paper

Tiny classifiers and tiny segmenters are the custom models created by me with the aim to have as few parameters as possible while still maintaining a reasonable accuracy.

Citation

@misc{glasses-detector,
  author = {Mantas Birškus},
  title = {Glasses Detector},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/mantasu/glasses-detector}},
  doi = {TBA}
}