The official PyTorch implementation of NITEC for eye contact detection.
Project description
NITEC: Versatile Hand-Annotated Eye Contact Dataset for Ego-Vision Interaction (Accepted at WACV24)
Citing
If you find our work useful, please cite the paper:
@misc{hempel2023nitec,
title={NITEC: Versatile Hand-Annotated Eye Contact Dataset for Ego-Vision Interaction},
author={Thorsten Hempel and Magnus Jung and Ahmed A. Abdelrahman and Ayoub Al-Hamadi},
year={2023},
eprint={2311.04505},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Paper
Abstract
Eye contact is a crucial non-verbal interaction modality and plays an important role in our everyday social life. While humans are very sensitive to eye contact, the capabilities of machines to capture a person's gaze are still mediocre. We tackle this challenge and present NITEC, a hand-annotated eye contact dataset for ego-vision interaction. NITEC exceeds existing datasets for ego-vision eye contact in size and variety of demographics, social contexts, and lighting conditions, making it a valuable resource for advancing ego-vision-based eye contact research. Our extensive evaluations on NITEC demonstrate strong cross-dataset performance, emphasizing its effectiveness and adaptability in various scenarios, that allows seamless utilization to the fields of computer vision, human-computer interaction, and social robotics. We make our NITEC dataset publicly available to foster reproducibility and further exploration in the field of ego-vision interaction.
Quick Usage:
pip install face_detection@git+https://github.com/elliottzheng/face-detection
pip install nitec
Example usage:
from nitec import NITEC_Classifier, visualize
import cv2
nitec_pipeline = NITEC_Classifier(
weights= CWD / 'models' / 'nitec_rs18_e20.pth',
device=torch.device('cuda') # or 'cpu'
)
cap = cv2.VideoCapture(0)
_, frame = cap.read()
# Process frame and visualize
results = nitec_pipeline.predict(frame)
frame = visualize(frame, results, confidence=0.5)
Train / Test
NITEC Dataset
Prepare the dataset as explained here.
Snapshots
Download from here: https://drive.google.com/drive/folders/1zc6NZZ6yA4NJ52Nn0bgky1XpZs9Z0hSJ?usp=sharing
Train
python train.py \
--gpu 0 \
--num_epochs 50 \
--batch_size 64 \
--lr 0.0001 \
Test
python test.py \
--snapshot models/nitec_rs18_20.pth \
--gpu 0 \
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.