SeetaPsych Gaze Follow

Gaze following estimation modules for SeetaPsych. Detects heads in scene and estimates where each person is looking (gaze point on image plane, and social gaze relations between people).

Usage

This project is already included in the seetapsych-lib default configuration. Download and use it via seetapsych-manager download.

For usage, refer to SeetaPsych.

Install optional algorithm dependencies:

uv pip install seetapsych-gaze-follow[all]

The algorithm modules automatically download model weights from ModelScope on first use, so the initial run may be slower due to model downloading.

You can additionally add this algorithm module using the following methods.

WebUI

Run seetapsych-webui with the --files argument to use it.

seetapsych-webui --files \
  seetapsych_gaze_follow/modules/head_detection.yml \
  seetapsych_gaze_follow/modules/cosi.yml

Programmatic Usage

Add the following code in your program to use this algorithm module.

from seetapsych_lib.runtime.factory import Factory
from seetapsych_lib.runtime.pipeline import Pipeline

factory = Factory()
factory.load_file_modules("seetapsych_gaze_follow/modules/head_detection.yml")
factory.load_file_modules("seetapsych_gaze_follow/modules/cosi.yml")

pipeline = Pipeline(factory, ...)

pipeline.add_attributes("head/detection", "head/gaze_point")
# Or for dyadic social gaze:
# pipeline.add_attributes('head/detection', 'head/social_gaze')

Module Catalog

Module Pipeline

The gaze following pipeline consists of two stages loaded from separate module configs:

1. HeadDetection (head_detection.yml) — multi-head detector, plus an optional HeadSelection post-processor.
2. CoSI (cosi.yml) — confidence-coordinated spatial integration model for gaze point and social gaze relation prediction.

Dependency graph:

• head/detection → head/gaze_point (single-person gaze following)
• head/detection → head/social_gaze (dyadic social gaze, requires ≥ 2 detected heads)

HeadDetection

Ultralytics multi-person head detector with pluggable selection/sorting post-process, used as the front-end for CoSI gaze-following models.

Module config: head_detection.yml

HeadDetection-CoSIGaze

Description: multi-person head detector with configurable confidence and NMS thresholds; produces head bounding boxes consumed by CoSI gaze-following and social-gaze packages.

Parameters

Models

HeadSelection

Description: Post-process that selects top-N head detections by size or confidence, then reorders them spatially (left-right/top-bottom) before passing to downstream gaze-following or social-gaze modules.

Parameters

Models: (none)

CoSIGaze

Confidence-coordinated Spatial Integration (CoSI) transformer for multi-person gaze following and dyadic social-gaze relation classification from a single RGB scene image.

Module config: cosi.yml

SceneGazeFollow-CoSIGaze

Description: Per-head scene-level gaze-following with CoSI transformer; for every input head box returns a 2D gaze target point (gaze_point_px) and a per-pixel gaze heatmap on the original scene image.

Parameters: (none)

Models

SocialGaze-CoSIGaze

Description: Dyadic social-gaze relation classifier using the shared CoSI transformer backbone; picks the top-2 most confident heads ordered horizontally (left = principal, right = associate) and predicts a 5-class relation, plus per-person gaze point and heatmap.

Parameters: (none)

Models

SocialGaze class mapping (social_gaze_id → social_gaze_label):