python-look 0.1.0

Simple wrapper code to run inference on images using the model and code from the Looking Repo from VITA-EPFL.

LOOK: A Wrapper For A Visual Attention Model

Simple wrapper code to run inference on images using the model and code from the Looking Repo from VITA-EPFL.

The Looking Repo is an official implementation of the paper Do pedestrians pay attention? Eye contact detection for autonomous driving

Image taken from : https://jooinn.com/people-walking-on-pedestrian-lane-during-daytime.html . Results obtained with the model trained on JackRabbot, Nuscenes, JAAD and Kitti. The model file is available at models/predictor and can be reused for testing with the predictor.

Table of contents

• Requirements
• Predictor
  • Example command
• Cite VITA-EPFL's work

Requirements

pip install python-look

Use 3.6.9 <= python < 3.9. Run pip3 install -r requirements.txt to get the dependencies

PLEASE DOWNLOAD THE MODEL HERE AND PLACE IN THE SAME LOCATION(or name the location when create the instance of the class)

Custom Training And Model Evaluation

Please refer to the original repo for training custom models and evaluation of models.

Predictor

Get predictions from the pretrained model using any image with the predictor API. The API extracts the human keypoints on the fly using OpenPifPaf. The predictor supports eye contact detection using human keypoints only.

Run Inference

• Create an instance of the Predictor (with default or custom settings)

  import look
  pred = look.Predictor()

• Run predict on an RGB OpenCV format image

  pred.predict(image)

• Call for results, in either data or an image with overlayed information

  output_image = pred.get_output_image()

Predictor Initialization Parameters

Parameter	Default Value	Description
transparency	0.4	transparency of the overlayed poses float
looking_threshold	0.5	eye contact threshold float
mode	joints	prediction mode string
device	'0'	CUDA device string
pifpaf_ver	shufflenetv2k30	PIFPAF ARG: backbone model to use string
model_path	models/predictor	To use custom trained model string
batch_size	1	PIFPAF ARG: processing batch size int
long_edge	None	PIFPAF ARG: rescale the long side of the image (aspect ratio maintained) int
loader_workers	None	PIFPAF ARG: number of workers for data loading int
disable_cuda	False	PIFPAF ARG: disable CUDA bool

Example Code To Run:

import cv2
import look

pred = look.Predictor()

cap = cv2.VideoCapture(0)

if cap.isOpened() == False:
    print("Camera feed is not open")
    exit()

width = int(cap.get(3))
height = int(cap.get(4))

print(f'Image Size: {width} X {height}')

while True:
    ret, frame = cap.read()

    if ret == True:
        pred.predict(frame)
        frame = pred.get_output_image()

        cv2.imshow('frame', frame)

        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
    
    else:
        print('Could not get frame from video')
        break

cap.release()
cv2.destroyAllWindows()

Credits

Credits to OpenPifPaf for the pose detection part.

Cite VITA-EPFL's work

If you use our work for your research please cite VITA-EPFL :)

@misc{belkada2021pedestrians,
      title={Do Pedestrians Pay Attention? Eye Contact Detection in the Wild}, 
      author={Younes Belkada and Lorenzo Bertoni and Romain Caristan and Taylor Mordan and Alexandre Alahi},
      year={2021},
      eprint={2112.04212},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}