SeetaPsych Gaze Follow

Gaze following estimation modules for SeetaPsych. Detects heads in scene and estimates where each person is looking (gaze point on image plane, and social gaze relations between people).

简体中文 | English

Usage

This project is already included in the seetapsych-lib default configuration. Download and use it via seetapsych-manager download.

For usage, refer to SeetaPsych.

Install optional algorithm dependencies:

uv pip install seetapsych-gaze-follow[all]

The algorithm modules automatically download model weights from ModelScope on first use, so the initial run may be slower due to model downloading.

You can additionally add this algorithm module using the following methods.

WebUI

Run seetapsych-webui with the --files argument to use it.

seetapsych-webui --files \
  seetapsych_gaze_follow/modules/head_detection.yml \
  seetapsych_gaze_follow/modules/cosi.yml

Programmatic Usage

Add the following code in your program to use this algorithm module.

from seetapsych_lib.runtime.factory import Factory
from seetapsych_lib.runtime.pipeline import Pipeline

factory = Factory()
factory.load_file_modules("seetapsych_gaze_follow/modules/head_detection.yml")
factory.load_file_modules("seetapsych_gaze_follow/modules/cosi.yml")

pipeline = Pipeline(factory, ...)

pipeline.add_attributes("head/detection", "head/gaze_point")
# Or for dyadic social gaze:
# pipeline.add_attributes('head/detection', 'head/social_gaze')

For complete end-to-end examples with visualization, see:

• examples/image_head_detection.py — static-image multi-head detection with confidence labels.
• examples/image_gaze_point.py — static-image per-head gaze following with per-head heatmap panels.
• examples/image_social_gaze.py — static-image dyadic social-gaze relation classification with per-person stacked panels.
• examples/video_social_gaze.py — offline video batch rendering of social gaze, preserving the original FPS and resolution.

Module Catalog

Module Pipeline

The gaze following pipeline consists of two stages loaded from separate module configs:

1. HeadDetection (head_detection.yml) — multi-head detector, plus an optional HeadSelection post-processor.
2. CoSI (cosi.yml) — confidence-coordinated spatial integration model for gaze point and social gaze relation prediction.

Dependency graph:

• head/detection → head/gaze_point (single-person gaze following)
• head/detection → head/social_gaze (dyadic social gaze, requires ≥ 2 detected heads)

HeadDetection

Ultralytics YOLO-based multi-person head detector with pluggable selection/sorting post-process, used as the front-end for CoSI gaze-following models.

Figure 1. HeadDetection output visualization — multi-head bounding boxes and per-box confidence scores.

Module config: head_detection.yml

HeadDetection-CoSIGaze

Description: Multi-person head detector with configurable confidence and NMS thresholds; produces head bounding boxes consumed by CoSI gaze-following and social-gaze packages.

Parameters

Models

Output Attributes

• head/detection — spec.

HeadSelection

Description: Post-process that selects top-N head detections by size or confidence, then reorders them spatially (left-right/top-bottom) before passing to downstream gaze-following or social-gaze modules.

Parameters

Models: (none)

Output Attributes

• head/selection — spec.
• head/detection — spec.

CoSIGaze

Confidence-coordinated Spatial Integration (CoSI) transformer for multi-person gaze following and dyadic social-gaze relation classification from a single RGB scene image.

Module config: cosi.yml

SceneGazeFollow-CoSIGaze

Figure 2. SceneGazeFollow output visualization — per-head stacked panels showing each head's gaze heatmap, gaze line and target point.

Description: Per-head scene-level gaze-following with CoSI transformer; for every input head box returns a 2D gaze target point (gaze_point_px) and a per-pixel gaze heatmap on the original scene image.

Parameters: (none)

Models

Output Attributes

• head/gaze_point — spec.

SocialGaze-CoSIGaze

Figure 3. SocialGaze output visualization — principal (green) and associate (red) panels, each with per-person heatmap, gaze target, and social-gaze relation label.

Description: Dyadic social-gaze relation classifier using the shared CoSI transformer backbone; picks the top-2 most confident heads ordered horizontally (left = principal, right = associate) and predicts a 5-class output of share, mutual, single, miss, void, plus per-person gaze point and heatmap.

Parameters: (none)

Models

Output Attributes

• head/social_gaze — spec.

SocialGaze class mapping (social_gaze_id → social_gaze_label):