space-stream 0.3.0

Send RGB-D images over spout / syphon / NDI with visiongraph.

Space Stream

Send RGB-D images over spout / syphon / NDI with visiongraph.

Source: Intel® RealSense™ Sample Data

Installation

It is recommended to use Python 3.11 (3.8, 3.9 or 3.10 should work too) and should run on any OS. First create a new virtualenv and activate it. After that install all dependencies:

pip install space-stream

ZED Camera

To be able to use a ZED Camera, please follow the tutorial on the ZED Python API website.

1. Install the ZED SDK (together with CUDA)
2. Run the command python "C:\Program Files (x86)\ZED SDK\get_python_api.py" inside the virtual python environment

Usage

Simply run the spacestream module with the following command to run a capturing pipeline (RealSense based). After that you can open a spout receiver / syphon receiver and check the result there.

space-stream --input realsense

To use the Azure Kinect use the azure input type:

space-stream --input azure

NDI

By default, either Syphon (MacOS) or Spout (Windows) is used to send the image to other applications. Since 0.3.0 it is possible to send the image over NDI by using the --ndi argument:

space-stream --input azure --ndi

OSC

To control the settings over OSC, start the application with the --osc argument. Please, listen for changes on port 7400 and to send changes, use port 7401 (by default).

space-stream --input azure --osc

/space-stream/record (Bidirectional): bool
/space-stream/codec (Bidirectional): DepthCodecType
/space-stream/min_distance (Bidirectional): float
/space-stream/max_distance (Bidirectional): float
/space-stream/depth_rectification (Bidirectional): bool
/space-stream/cam_auto_exposure (Bidirectional): bool
/space-stream/cam_exposure (Bidirectional): int
/space-stream/cam_iso (Bidirectional): int
/space-stream/cam_auto_white_balance (Bidirectional): bool
/space-stream/cam_white_balance (Bidirectional): int
/space-stream/masking (Bidirectional): bool
/space-stream/stream_name (Bidirectional): str

Build

python setup.py distribute

Development

To develop with this project, clone the git repository and install the dependencies from the requirements:

pip install -r requirements.txt

To call the module directly, use the -m command from python:

python -m spacestream

Depth Codec

By default the depthmap is encoded by the linear codec. It is possible to change the behaviour to use a specific encoding method. Be aware that some functions have an impact on performance. Here is a list of all available codecs:

Linear,
UniformHue
InverseHue

The codecs UniformHue and InverseHue are implemented according to the Intel whitepaper about Depth image compression by colorization.

Bit Depth

The encoded bit-depth depends on the codec used. For Linear codec there are two different bit-depths encoded. First the 8-bit encoding in the red channel and 16-bit encoded values in the green (MSB) and blue (LSB) channel.

Distance Range

To define the min and max distance to encode, use the --min-distance and --max-distance parameter.

Help

usage: space-stream [-h] [-c CONFIG] [-s SETTINGS]
                    [--loglevel {critical,error,warning,info,debug}]
                    [--record RECORD]
                    [--codec Linear, UniformHue, InverseHue, RSColorizer]
                    [--min-distance MIN_DISTANCE]
                    [--max-distance MAX_DISTANCE] [--stream-name STREAM_NAME]
                    [--input video-capture,image,realsense,azure,camgear,zed]
                    [--input-size width height] [--input-fps INPUT_FPS]
                    [--input-rotate 90,-90,180] [--input-flip h,v]
                    [--input-mask INPUT_MASK] [--input-crop x y width height]
                    [--raw-input] [--channel CHANNEL]
                    [--input-skip INPUT_SKIP]
                    [--input-backend any,vfw,v4l,v4l2,firewire,fireware,ieee1394,dc1394,cmu1394,qt,unicap,dshow,pvapi,openni,openni_asus,android,xiapi,avfoundation,giganetix,msmf,winrt,intelperc,openni2,openni2_asus,gphoto2,gstreamer,ffmpeg,images,aravis,opencv_mjpeg,intel_mfx,xine]
                    [-src SOURCE] [--input-path INPUT_PATH]
                    [--input-delay INPUT_DELAY] [--exposure EXPOSURE]
                    [--gain GAIN] [--white-balance WHITE_BALANCE] [--depth]
                    [--depth-as-input] [-ir] [--rs-serial RS_SERIAL]
                    [--rs-json RS_JSON] [--rs-play-bag RS_PLAY_BAG]
                    [--rs-record-bag RS_RECORD_BAG] [--rs-disable-emitter]
                    [--rs-bag-offline]
                    [--rs-auto-exposure-limit RS_AUTO_EXPOSURE_LIMIT]
                    [--rs-auto-gain-limit RS_AUTO_GAIN_LIMIT]
                    [--rs-filter decimation,spatial,temporal,hole-filling [decimation,spatial,temporal,hole-filling ...]]
                    [--rs-color-scheme Jet,Classic,WhiteToBlack,BlackToWhite,Bio,Cold,Warm,Quantized,Pattern]
                    [--k4a-align-to-color] [--k4a-align-to-depth]
                    [--k4a-device K4A_DEVICE] [--k4a-depth-clipping min max]
                    [--k4a-ir-clipping min max] [--k4a-play-mkv K4A_PLAY_MKV]
                    [--k4a-record-mkv K4A_RECORD_MKV]
                    [--k4a-depth-mode OFF,NFOV_2X2BINNED,NFOV_UNBINNED,WFOV_2X2BINNED,WFOV_UNBINNED,PASSIVE_IR]
                    [--k4a-passive-ir]
                    [--k4a-color-resolution OFF,RES_720P,RES_1080P,RES_1440P,RES_1536P,RES_2160P,RES_3072P]
                    [--k4a-color-format COLOR_MJPG,COLOR_NV12,COLOR_YUY2,COLOR_BGRA32,DEPTH16,IR16,CUSTOM8,CUSTOM16,CUSTOM]
                    [--k4a-wired-sync-mode STANDALONE,MASTER,SUBORDINATE]
                    [--k4a-subordinate-delay-off-master-usec K4A_SUBORDINATE_DELAY_OFF_MASTER_USEC]
                    [--midas] [--mask]
                    [--segnet mediapipe,mediapipe-light,mediapipe-heavy]
                    [--parallel] [--num-threads NUM_THREADS] [--no-fastmath]
                    [--no-filter] [--no-preview] [--record-crf RECORD_CRF]
                    [--view-pcd] [--view-3d] [--osc] [--osc-host OSC_HOST]
                    [--osc-in-port OSC_IN_PORT] [--osc-out-port OSC_OUT_PORT]
                    [--ndi]

RGB-D framebuffer sharing demo for visiongraph.

options:
  -h, --help            show this help message and exit
  -c CONFIG, --config CONFIG
                        Configuration file path.
  -s SETTINGS, --settings SETTINGS
                        Settings file path (json).
  --loglevel {critical,error,warning,info,debug}
                        Provide logging level. Example --loglevel debug,
                        default=warning
  --record RECORD       Record output into recordings folder.
  --codec Linear, UniformHue, InverseHue, RSColorizer
                        Codec how the depth map will be encoded.
  --min-distance MIN_DISTANCE
                        Min distance to perceive by the camera.
  --max-distance MAX_DISTANCE
                        Max distance to perceive by the camera.
  --stream-name STREAM_NAME
                        Spout / Syphon stream name.

input provider:
  --input video-capture,image,realsense,azure,camgear,zed
                        Image input provider, default: video-capture.
  --input-size width height
                        Requested input media size.
  --input-fps INPUT_FPS
                        Requested input media framerate.
  --input-rotate 90,-90,180
                        Rotate input media.
  --input-flip h,v      Flip input media.
  --input-mask INPUT_MASK
                        Path to the input mask.
  --input-crop x y width height
                        Crop input image.
  --raw-input           Skip automatic input conversion to 3-channel image.
  --channel CHANNEL     Input device channel (camera id, video path, image
                        sequence).
  --input-skip INPUT_SKIP
                        If set the input will be skipped to the value in
                        milliseconds.
  --input-backend any,vfw,v4l,v4l2,firewire,fireware,ieee1394,dc1394,cmu1394,qt,unicap,dshow,pvapi,openni,openni_asus,android,xiapi,avfoundation,giganetix,msmf,winrt,intelperc,openni2,openni2_asus,gphoto2,gstreamer,ffmpeg,images,aravis,opencv_mjpeg,intel_mfx,xine
                        VideoCapture API backends identifier., default: any.
  -src SOURCE, --source SOURCE
                        Generic input source for all inputs.
  --input-path INPUT_PATH
                        Path to the input image.
  --input-delay INPUT_DELAY
                        Input delay time (s).
  --exposure EXPOSURE   Exposure value (usec) for depth camera input (disables
                        auto-exposure).
  --gain GAIN           Gain value for depth input (disables auto-exposure).
  --white-balance WHITE_BALANCE
                        White-Balance value for depth input (disables auto-
                        white-balance).
  --depth               Enable RealSense depth stream.
  --depth-as-input      Use colored depth stream as input stream.
  -ir, --infrared       Use infrared as input stream.
  --rs-serial RS_SERIAL
                        RealSense serial number to choose specific device.
  --rs-json RS_JSON     RealSense json configuration to apply.
  --rs-play-bag RS_PLAY_BAG
                        Path to a pre-recorded bag file for playback.
  --rs-record-bag RS_RECORD_BAG
                        Path to a bag file to store the current recording.
  --rs-disable-emitter  Disable RealSense IR emitter.
  --rs-bag-offline      Disable realtime bag playback.
  --rs-auto-exposure-limit RS_AUTO_EXPOSURE_LIMIT
                        Auto exposure limit (ms).
  --rs-auto-gain-limit RS_AUTO_GAIN_LIMIT
                        Auto gain limit (16-248).
  --rs-filter decimation,spatial,temporal,hole-filling [decimation,spatial,temporal,hole-filling ...]
                        RealSense depth filter.
  --rs-color-scheme Jet,Classic,WhiteToBlack,BlackToWhite,Bio,Cold,Warm,Quantized,Pattern
                        Color scheme for depth map, default: WhiteToBlack.
  --k4a-align-to-color  Align azure frames to color frame.
  --k4a-align-to-depth  Align azure frames to depth frame.
  --k4a-device K4A_DEVICE
                        Azure device id.
  --k4a-depth-clipping min max
                        Depth input clipping.
  --k4a-ir-clipping min max
                        Infrared input clipping.
  --k4a-play-mkv K4A_PLAY_MKV
                        Path to a pre-recorded bag file for playback.
  --k4a-record-mkv K4A_RECORD_MKV
                        Path to a mkv file to store the current recording.
  --k4a-depth-mode OFF,NFOV_2X2BINNED,NFOV_UNBINNED,WFOV_2X2BINNED,WFOV_UNBINNED,PASSIVE_IR
                        Azure depth mode, default: NFOV_UNBINNED.
  --k4a-passive-ir      Use passive IR input.
  --k4a-color-resolution OFF,RES_720P,RES_1080P,RES_1440P,RES_1536P,RES_2160P,RES_3072P
                        Azure color resolution (overwrites input-size),
                        default: RES_720P.
  --k4a-color-format COLOR_MJPG,COLOR_NV12,COLOR_YUY2,COLOR_BGRA32,DEPTH16,IR16,CUSTOM8,CUSTOM16,CUSTOM
                        Azure color image format, default: COLOR_BGRA32.
  --k4a-wired-sync-mode STANDALONE,MASTER,SUBORDINATE
                        Synchronization mode when connecting two or more
                        devices together, default: STANDALONE.
  --k4a-subordinate-delay-off-master-usec K4A_SUBORDINATE_DELAY_OFF_MASTER_USEC
                        The external synchronization timing.
  --midas               Use midas for depth capture.

masking:
  --mask                Apply mask by segmentation algorithm.
  --segnet mediapipe,mediapipe-light,mediapipe-heavy
                        Segmentation Network, default: mediapipe.

performance:
  --parallel            Enable parallel for codec operations.
  --num-threads NUM_THREADS
                        Number of threads for parallelization.
  --no-fastmath         Disable fastmath for codec operations.

debug:
  --no-filter           Disable realsense image filter.
  --no-preview          Disable preview to speed.
  --record-crf RECORD_CRF
                        Recording compression rate.
  --view-pcd            Display PCB preview (deprecated, use --view-3d).
  --view-3d             Display PCB preview.
  
osc:
  --osc                 Enable OSC support for settings.
  --osc-host OSC_HOST   OSC host address (default: 0.0.0.0)
  --osc-in-port OSC_IN_PORT
                        OSC receiving port address (default: 7401)
  --osc-out-port OSC_OUT_PORT
                        OSC receiving port address (default: 7400)

output:
  --ndi                 Use NDI for frame buffer sharing.

Args that start with '--' can also be set in a config file (specified via -c).
Config file syntax allows: key=value, flag=true, stuff=[a,b,c] (for details,
see syntax at https://goo.gl/R74nmi). In general, command-line values override
config file values which override defaults.

About

Copyright (c) 2024 Florian Bruggisser