Whitebox Plugin - VideoJS

This is a plugin for whitebox that provides VideoJS integration, including WHEP (WebRTC) streaming support.

Installation

Simply install the plugin to whitebox:

poetry add whitebox-plugin-videojs

Frontend development

In a prepared core backend dev container, use make test-frontend for the selected source suite. Ordinary Python archives contain JSX; core explicitly assembles the installed separate remote. Follow the shared frontend package guide and JSX testing guide for prerequisites, assembly/restart lifecycle and private test setup.

Video.js 8.16.1 remains a managed JS/CSS asset. Bootstrap must load it before the package's whep-tech.js and before mounting ui.video. No npm Video.js copy is bundled. Source tests use a player double. Use the owner application gate to build/install your checkout's wheel for controlled playback checks. Aggregate application checks use installed image packages, not local plugin edits; that guide lists prerequisites. Native decode/local WHEP are not physical-camera or production-streaming proof.

WHEP Tech - WebRTC Streaming

This plugin includes a custom Video.js Tech for WHEP (WebRTC-HTTP Egress Protocol) streaming. WHEP enables ultra-low-latency video playback from WebRTC-capable media servers.

What is WHEP?

WHEP (WebRTC-HTTP Egress Protocol) is a standardized protocol for receiving WebRTC streams. It simplifies WebRTC consumption by using HTTP for signaling:

1. Client sends an SDP offer via HTTP POST to the WHEP endpoint
2. Server responds with an SDP answer
3. ICE candidates are exchanged (trickle ICE supported)
4. WebRTC peer connection is established
5. Media flows over WebRTC (UDP/SRTP)

Unlike traditional WebRTC which requires custom signaling servers, WHEP uses simple HTTP - making it easy to integrate with any media server that supports it.

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────────┐
│                           STREAMING PIPELINE                                │
└─────────────────────────────────────────────────────────────────────────────┘

┌──────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Insta360   │     │     FFmpeg      │     │   SRS Server    │
│    Camera    │────▶│   (transcode)   │────▶│  (media server) │
│              │WiFi │                 │RTMP │                 │
└──────────────┘     │ • Dual fisheye  │     │ • RTMP ingest   │
                     │   → equirect    │     │ • WebRTC/WHEP   │
                     │ • H.264 encode  │     │   egress        │
                     └─────────────────┘     └────────┬────────┘
                                                      │
                                                      │ WHEP (HTTP + WebRTC)
                                                      ▼
┌──────────────────────────────────────────────────────────────────────────────┐
│                              BROWSER                                         │
│  ┌─────────────────┐     ┌─────────────────┐     ┌──────────────────┐        │
│  │    WHEP Tech    │     │    Video.js     │     │    Pannellum     │        │
│  │  (whep-tech.js) │────▶│     Player      │────▶│   360 Viewer     │        │
│  │                 │     │                 │     │                  │        │
│  │ • SDP exchange  │     │ • <video> with  │     │ • WebGL canvas   │        │
│  │ • ICE handling  │     │   srcObject     │     │ • Equirectangular│        │
│  │ • Reconnection  │     │ • Playback ctrl │     │   projection     │        │
│  └─────────────────┘     └─────────────────┘     └──────────────────┘        │
└──────────────────────────────────────────────────────────────────────────────┘

Key Components

1. WHEP Tech (whep-tech.js)

Custom Video.js Tech that handles WebRTC playback:

// Usage
const player = videojs('my-video', {
  techOrder: ['whep'],  // Use WHEP tech
  sources: [{
    src: 'https://srs-server/rtc/v1/whep/?app=live&stream=livestream',
    type: 'application/whep'  // MIME type triggers WHEP tech
  }]
});

Features:

• Automatic SDP offer/answer exchange
• Trickle ICE candidate handling
• Connection state management with events
• Auto-reconnection with exponential backoff
• Graceful cleanup on dispose

Events emitted:

• whep:connecting - Starting connection
• whep:streaming - Media flowing
• whep:reconnecting - Connection lost, retrying
• whep:error - Connection failed

2. Integration with Pannellum (360 Video)

The whitebox-plugin-pannellum provides 360-degree video rendering. It detects WebRTC streams via srcObject:

// In Pannellum360Video.jsx
const isWebRTC = !!videoEl?.srcObject;

if (isWebRTC) {
  // Direct Pannellum initialization - pass video element as texture
  pannellum.viewer(container, {
    type: "equirectangular",
    panorama: videoEl,  // Video element with srcObject
    dynamic: true,
  });
} else {
  // Regular video URL - use videojs-pannellum plugin
  player.pannellum(config);
}

Why the distinction?

• WebRTC uses srcObject (MediaStream) not src (URL)
• The videojs-pannellum plugin doesn't handle srcObject
• Direct Pannellum init passes the video element as a WebGL texture source

Connection Flow

1. Player initialized with WHEP source
   └─▶ WhepTech.setSrc() called

2. Create RTCPeerConnection
   └─▶ Add transceiver for video (recvonly)
   └─▶ Create SDP offer

3. HTTP POST offer to WHEP endpoint
   └─▶ Server returns SDP answer
   └─▶ setRemoteDescription(answer)

4. ICE candidate gathering
   └─▶ Local candidates sent via PATCH
   └─▶ Remote candidates from Link headers

5. Connection established
   └─▶ ontrack fires with MediaStream
   └─▶ video.srcObject = stream
   └─▶ Emit 'whep:streaming'

6. On disconnect
   └─▶ Exponential backoff retry
   └─▶ Emit 'whep:reconnecting'

Frame Detection for 360 Video

WebRTC streams require special handling for 360 rendering. We must wait for actual video frames before initializing Pannellum:

// waitForVideoFrame() in Pannellum360Video.jsx
if ("requestVideoFrameCallback" in videoEl) {
  // Chrome 83+, Firefox 130+ - fires when frame is ready
  videoEl.requestVideoFrameCallback(checkFrame);
} else {
  // Fallback: poll videoWidth/videoHeight
  setInterval(checkDimensions, 100);
}

This prevents the "black sphere" issue where Pannellum initializes before any video data arrives.

Additional Instructions

• Plugin Development Guide
• Plugin Testing Guide
• Contributing Guidelines