SeetaPsych Attributes

Face and body based psychology analysis

SeetaPsych Lib is a Python library for face- and body-based psychology analysis. It provides a modular Pipeline/Runner runtime and an optional Streamlit WebUI.

This project is used to manage the specifications for various attribute outputs, providing a unified standard so that different algorithm implementations can produce interchangeable and reusable module outputs.

TypedDict Type Hints

Alongside the JSON schemas documented below, this project ships a set of ready-to-use TypedDict declarations under seetapsych_attributes.types so that your IDE can provide auto-completions and static type checks directly on the runner's report dict:

# -*- coding: utf-8 -*-
import json

import cv2

from seetapsych_lib.runtime.factory import Factory
from seetapsych_lib.runtime.pipeline import Pipeline
from seetapsych_lib.runtime.runner import Runner

from seetapsych_attributes.types import Report, BBox, FaceDetection


def main():
    factory = Factory()
    factory.load_builtin_modules()

    pipeline = Pipeline(factory, attributes=["face/detection"])
    pipeline.solve()
    pipeline.install_requirements()
    pipeline.cache_models()

    runner = Runner(pipeline)

    report: Report = runner.run(data={"default": cv2.imread("data/a.jpg")})

    # IDE autocompletion + type inference for every attribute key:
    detections: FaceDetection | None = report.get("face_detection")
    if detections:
        first: BBox = detections[0]
        x1, y1, x2, y2 = first["xyxy"]
        score: float = first["score"]
        print(f"face at ({x1},{y1})-({x2},{y2}), score = {score:.3f}")

    print(json.dumps(report, indent=2, ensure_ascii=False))


if __name__ == "__main__":
    main()

The top-level Report TypedDict includes every attribute key defined in the catalog (all fields are optional, because a pipeline may only request a subset). Per-attribute element types such as BBox, Landmarks, Selection, ActionUnits, Expression, HeartRate, HeadSocialGaze, etc. are also exported individually.

Catalog

• face/detection Face detection results as rectangular bounding boxes.
• face/landmarks Facial landmarks for basic alignment: L-eye, R-eye, nose, L-mouth, R-mouth (10 interleaved floats).
• face/selection Selected face PID. Selected face order is reflected in face/detection and face/landmarks.
• face/action_units Indicate the confidence level of each Action Unit. Not all Action Units' results may be output.
• face/expression Indicate the confidence level of each expression.
• face/dense_landmarks 280-point dense facial landmarks (560 interleaved [x,y] floats).
• face/mesh 468-point 3D face mesh landmarks in normalized coordinates.
• face/gaze_screen Per-eye screen-space point-of-gaze coordinates in pixels and camera-space point-of-gaze coordinates in millimeters. Camera-space coordinates may be 2D or 3D depending on the algorithm; origin is at the camera center.
• face/heart_rate Heart rate (BPM) estimated from buffered face video frames.
• face/dimensional_affect Continuous valence-arousal affect dimensions alongside discrete expressions and Action Units.
• head/detection Multi-person head bounding box detection results.
• head/selection Top-N head selection result (count + original indices), reordering head_detection.
• head/gaze_point Per-head 2D scene gaze target point with associated likelihood heatmap.
• head/social_gaze Dyadic social gaze relations between two detected people. Class set: share, mutual, single, miss, void.

face/detection

Face detection results as rectangular bounding boxes.

Properties

• face_detection (array, required)
  • Items: Refer to BBox.

Definitions

• BBox (object)
  • xyxy (array, required): Length must be equal to 4.
    • Items (number)
  • score (number, required)

Examples

{
    "face_detection": [
        {
            "score": 0.5,
            "xyxy": [
                100,
                200,
                300,
                400
            ]
        }
    ]
}

face/landmarks

Facial landmarks for basic alignment: L-eye, R-eye, nose, L-mouth, R-mouth (10 interleaved floats).

Properties

• face_landmarks (array, required)
  • Items: Refer to Landmarks.

Definitions

• Landmarks (object)
  • landmarks (array, required): Length must be equal to 10.
    • Items (number)

Examples

{
    "face_landmarks": [
        {
            "landmarks": [
                100,
                100,
                200,
                200,
                300,
                300,
                400,
                400,
                500,
                500
            ]
        }
    ]
}

face/selection

Selected face PID. Selected face order is reflected in face/detection and face/landmarks.

Properties

• face_selection (required): Refer to Selection.

Definitions

• Selection (object)
  • pid (integer, required): PID of selected face detection (1-based).

Examples

{
    "face_detection": [
        {
            "score": 0.5,
            "xyxy": [
                100,
                200,
                300,
                400
            ]
        }
    ],
    "face_selection": {
        "pid": 1
    }
}

face/action_units

Indicate the confidence level of each Action Unit. Not all Action Units' results may be output.

Properties

• face_action_units (array, required)
  • Items: Refer to ActionUnits.

Definitions

• ActionUnits (object)
  • AU1: [0, 1]. Inner Brow Raiser. Default: null.
    • Any of
      • number
      • null
  • AU2: [0, 1]. Outer Brow Raiser. Default: null.
    • Any of
      • number
      • null
  • AU4: [0, 1]. Brow Lowerer. Default: null.
    • Any of
      • number
      • null
  • AU5: [0, 1]. Upper Lid Raiser. Default: null.
    • Any of
      • number
      • null
  • AU6: [0, 1]. Cheek Raiser. Default: null.
    • Any of
      • number
      • null
  • AU7: [0, 1]. Lid Tightener. Default: null.
    • Any of
      • number
      • null
  • AU9: [0, 1]. Nose Wrinkler. Default: null.
    • Any of
      • number
      • null
  • AU10: [0, 1]. Upper Lip Raiser. Default: null.
    • Any of
      • number
      • null
  • AU12: [0, 1]. Lip Corner Puller. Default: null.
    • Any of
      • number
      • null
  • AU15: [0, 1]. Lip Corner Depressor. Default: null.
    • Any of
      • number
      • null
  • AU17: [0, 1]. Chin Raiser. Default: null.
    • Any of
      • number
      • null
  • AU20: [0, 1]. Lip Stretcher. Default: null.
    • Any of
      • number
      • null
  • AU23: [0, 1]. Lip Tightener. Default: null.
    • Any of
      • number
      • null
  • AU24: [0, 1]. Lip Pressor. Default: null.
    • Any of
      • number
      • null
  • AU25: [0, 1]. Lips Part. Default: null.
    • Any of
      • number
      • null
  • AU26: [0, 1]. Jaw Drop. Default: null.
    • Any of
      • number
      • null

Examples

{
    "face_action_units": [
        {
            "AU1": 0.5,
            "AU10": 0.5,
            "AU12": 0.5,
            "AU15": 0.5,
            "AU17": 0.5,
            "AU2": 0.5,
            "AU20": 0.5,
            "AU23": 0.5,
            "AU24": 0.5,
            "AU25": 0.5,
            "AU26": 0.5,
            "AU4": 0.5,
            "AU5": 0.5,
            "AU6": 0.5,
            "AU7": 0.5,
            "AU9": 0.5
        }
    ]
}

face/expression

Indicate the confidence level of each expression.

Properties

• face_expression (array, required)
  • Items: Refer to Expression.

Definitions

• Expression (object)
  • neutral: Confidence in [0, 1]. Default: null.
    • Any of
      • number
      • null
  • anger: Confidence in [0, 1]. Default: null.
    • Any of
      • number
      • null
  • disgust: Confidence in [0, 1]. Default: null.
    • Any of
      • number
      • null
  • fear: Confidence in [0, 1]. Default: null.
    • Any of
      • number
      • null
  • happy: Confidence in [0, 1]. Default: null.
    • Any of
      • number
      • null
  • sad: Confidence in [0, 1]. Default: null.
    • Any of
      • number
      • null
  • surprise: Confidence in [0, 1]. Default: null.
    • Any of
      • number
      • null

Examples

{
    "face_expression": [
        {
            "anger": 0.01,
            "disgust": 0.01,
            "fear": 0.01,
            "happy": 0.94,
            "neutral": 0.01,
            "sad": 0.01,
            "surprise": 0.01
        }
    ]
}

face/dense_landmarks

280-point dense facial landmarks (560 interleaved [x,y] floats).

Properties

• face_dense_landmarks (array, required)
  • Items: Refer to DenseLandmarks.

Definitions

• DenseLandmarks (object)
  • landmarks (array, required): Length must be equal to 560.
    • Items (number)

Examples

{
    "face_dense_landmarks": [
        {
            "landmarks": "[100.0] * 560"
        }
    ]
}

face/mesh

468-point 3D face mesh landmarks in normalized coordinates.

Properties

• face_mesh (array, required)
  • Items: Refer to MeshLandmarks.

Definitions

• MeshLandmarks (object)
  • normalized_3d_landmarks (array, required): Length must be equal to 1404.
    • Items (number)

Examples

{
    "face_mesh": [
        {
            "normalized_3d_landmarks": "[0.5] * 1404"
        }
    ]
}

face/gaze_screen

Per-eye screen-space point-of-gaze coordinates in pixels and camera-space point-of-gaze coordinates in millimeters. Camera-space coordinates may be 2D or 3D depending on the algorithm; origin is at the camera center.

Properties

• face_gaze_screen (array, required)
  • Items: Refer to GazeScreen.

Definitions

• GazeData (object): Gaze estimation result for a single face.
  • success (boolean, required): Whether gaze estimation succeeded for this face.
  • gaze_screen_px (required): Per-eye point-of-gaze coordinates in screen-space pixels. The origin is at the left top corner of the screen. The coordinate system is shown as Fig. 1. Refer to GazePoint.
  • gaze_camera_mm (required): Per-eye point-of-gaze coordinates in the camera coordinate system, measured in millimeters. The coordinate origin is located at the camera optical center. Depending on the gaze estimation algorithm, the point-of-gaze can be represented either as a 2D coordinate on the camera image plane or as a 3D point in camera space. The coordinate definitions for 2D and 3D gaze_camera_mm are illustrated in Fig. 2 and Fig. 3, respectively. Different algorithms may natively output in different coordinate frames; all values are normalized to the conventions documented here before being returned. Refer to GazePoint.
• GazePoint (object): Point-of-gaze coordinates for two eyes in either screen-pixel or camera-millimeter space.
  • left_eye (array, required): Gaze-screen-px [x,y] or gaze-camera-mm [x,y(,z)]; 2D or 3D depending on algorithm; empty list if unavailable. Length must be between 0 and 3 (inclusive).
    • Items (number)
  • right_eye (array, required): Screen-px [x,y] or camera-mm [x,y(,z)]; 2D or 3D depending on algorithm; empty list if unavailable. Length must be between 0 and 3 (inclusive).
    • Items (number)
• GazeScreen (object): Wrapper for per-face gaze data in face/gaze_screen schema.
  • gaze (required): Refer to GazeData.

Figure 1. Coordinate system for gaze_screen_px

Figure 2. Coordinate system for gaze_camera_mm(2D)

Figure 3. Coordinate system for gaze_camera_mm(3D)

Examples

{
    "face_gaze_screen": [
        {
            "gaze": {
                "gaze_camera_mm": {
                    "left_eye": [
                        155.0,
                        50.0,
                        25.0
                    ],
                    "right_eye": [
                        155.0,
                        50.0,
                        25.0
                    ]
                },
                "gaze_screen_px": {
                    "left_eye": [
                        960.0,
                        540.0
                    ],
                    "right_eye": [
                        960.0,
                        540.0
                    ]
                },
                "success": true
            }
        }
    ]
}

face/heart_rate

Heart rate (BPM) estimated from buffered face video frames.

Properties

• face_heart_rate (required): Refer to HeartRate.

Definitions

• HeartRate (object)
  • fps (number, required): Current estimated frames per second.
  • wait_seconds (number, required): Seconds remaining until enough data is buffered. 0.0 when HR is ready.
  • hr_bpm: Estimated heart rate in beats per minute. Present only when ready. Default: null.
    • Any of
      • number
      • null

Examples

{
    "face_heart_rate": {
        "fps": 30.0,
        "hr_bpm": 72.5,
        "wait_seconds": 0.0
    }
}

{
    "face_heart_rate": {
        "fps": 30.0,
        "wait_seconds": 5.2
    }
}

face/dimensional_affect

Continuous valence-arousal affect dimensions alongside discrete expressions and Action Units.

Properties

• face_dimensional_affect (array, required)
  • Items: Refer to DimensionalAffect.

Definitions

• DimensionalAffect (object)
  • valence (number, required): Valence dimension in continuous affect space. Positive = pleasant, negative = unpleasant.
  • arousal (number, required): Arousal dimension in continuous affect space. Positive = activated, negative = calm.

Examples

{
    "face_dimensional_affect": [
        {
            "arousal": 0.32,
            "valence": 0.85
        }
    ]
}

head/detection

Multi-person head bounding box detection results.

Properties

• head_detection (array, required)
  • Items: Refer to HeadBBox.

Definitions

• HeadBBox (object)
  • xyxy (array, required): Length must be equal to 4.
    • Items (integer)
  • score (number, required)

Examples

{
    "head_detection": [
        {
            "score": 0.85,
            "xyxy": [
                100,
                200,
                300,
                400
            ]
        }
    ]
}

head/selection

Top-N head selection result (count + original indices), reordering head_detection.

Properties

• head_selection (required): Refer to HeadSelection.

Definitions

• HeadSelection (object)
  • count (integer, required): Number of selected head detections.
  • selected_indices (array, required): Indices of selected detections in the original head_detection list, before sorting.
    • Items (integer)

Examples

{
    "head_detection": [
        {
            "score": 0.85,
            "xyxy": [
                100,
                200,
                300,
                400
            ]
        }
    ],
    "head_selection": {
        "count": 1,
        "selected_indices": [
            0
        ]
    }
}

head/gaze_point

Per-head 2D scene gaze target point with associated likelihood heatmap.

Properties

• head_gaze_point (array, required)
  • Items: Refer to HeadGazePoint.

Definitions

• HeadGazePoint (object)
  • head_location_xyxy (array, required): Length must be equal to 4.
    • Items (integer)
  • gaze_point_px (array, required): Length must be equal to 2.
    • Items (number)
  • heatmap (array, required): 2D gaze likelihood heatmap over the scene. Runtime type: numpy.ndarray of float32, shape [image_height, image_width], values in [0, 1] probability range.
    • Items (array)
      • Items (number)

Examples

{
    "head_gaze_point": [
        {
            "gaze_point_px": [
                640.0,
                360.0
            ],
            "head_location_xyxy": [
                100,
                200,
                300,
                400
            ],
            "heatmap": "numpy.ndarray(shape=[H, W], dtype=float32) -- 2D [0,1] gaze likelihood heatmap"
        }
    ]
}

head/social_gaze

Dyadic social gaze relations between two detected people. Class set: share, mutual, single, miss, void.

Properties

• head_social_gaze (required): Refer to HeadSocialGaze.

Definitions

• HeadSocialGaze (object)
  • principal: Left-side / primary person in dyadic interaction. Default: null.
    • Any of
      • : Refer to SocialGazePerson.
      • null
  • associate: Right-side / secondary person in dyadic interaction. Default: null.
    • Any of
      • : Refer to SocialGazePerson.
      • null
  • success (boolean): Whether at least two heads were detected for social gaze inference. Default: true.
• SocialGazePerson (object)
  • head_location_xyxy (array, required): Length must be equal to 4.
    • Items (integer)
  • gaze_point_px (array, required): Length must be equal to 2.
    • Items (number)
  • heatmap (array, required): 2D gaze likelihood heatmap. Runtime type: numpy.ndarray of float32, shape [image_height, image_width], values in [0, 1] probability range.
    • Items (array)
      • Items (number)
  • social_gaze_id (integer, required): Integer class ID of the social gaze relation. Ordered mapping: 0=share, 1=mutual, 2=single, 3=miss, 4=void.
  • social_gaze_label (string, required): Human-readable social gaze relation label. Possible values: share, mutual, single, miss, void. Index of the value matches social_gaze_id.

Examples

{
    "head_social_gaze": {
        "associate": {
            "gaze_point_px": [
                200.0,
                300.0
            ],
            "head_location_xyxy": [
                600,
                200,
                800,
                400
            ],
            "heatmap": "numpy.ndarray(shape=[H, W], dtype=float32) -- 2D [0,1] gaze likelihood heatmap",
            "social_gaze_id": 1,
            "social_gaze_label": "mutual"
        },
        "principal": {
            "gaze_point_px": [
                800.0,
                300.0
            ],
            "head_location_xyxy": [
                100,
                200,
                300,
                400
            ],
            "heatmap": "numpy.ndarray(shape=[H, W], dtype=float32) -- 2D [0,1] gaze likelihood heatmap",
            "social_gaze_id": 1,
            "social_gaze_label": "mutual"
        },
        "success": true
    }
}

{
    "head_social_gaze": {
        "success": false
    }
}