rgbd-pd-viewer 0.1.0

Open3D GUI viewer for RGB-D frames with point picking and Robot Base Marker for camera-to-world calibration.

rgbd-pd-viewer

An interactive Open3D GUI viewer for RGB-D data. Loads a single (depth, RGB) pair or a synchronized directory of pairs, back-projects them into a coloured point cloud, and exposes:

• frame navigation (Prev / Next / jump-to-frame)
• camera controls (azimuth / elevation / distance sliders + free mouse)
• point picking — Shift + left-click to read camera-frame XYZ
• Robot Base Marker — place a sphere at the picked point, nudge it onto the robot base, rotate the world frame, and save the resulting T_wc / T_cw to JSON (handy for hand-eye / camera-to-world calibration)
• a camera-origin frustum so the data-camera location is always visible
• adding extra (depth, RGB) sources on the fly, each with its own affine depth correction depth_new = A * depth + B

The viewer assumes inputs land at 480 × 640 after a centre-crop + resize; if your raw depth/RGB has a different aspect ratio the loader crops to match (never stretches), then resizes.

---

Installation

pip install rgbd-pd-viewer

Or, from a checkout:

git clone https://github.com/yuzhenchen/rgbd-pd-viewer.git
cd rgbd-pd-viewer
pip install -e .

Note on Open3D. On Linux the Open3D GUI needs an OpenGL-capable display. Over SSH use ssh -X (or VirtualGL); inside Docker you'll need to forward $DISPLAY and mount /tmp/.X11-unix.

---

Quick start

There are two ways to load data.

Mode 1 — pass everything on the command line (single frame)

Positional order is fixed: RGB DEPTH [INTRINSICS].

rgbd-pd-viewer color.png depth.npy 640,640,320,240

• RGB — path to .jpg / .png / .bmp / .tiff
• DEPTH — path to .npy / .npz
• INTRINSICS — optional, four comma-separated floats fx,fy,cx,cy (in pixels, in that exact order). No spaces. Default if omitted: 640,640,320,240.

So the shortest form is just:

rgbd-pd-viewer color.png depth.npy

Mode 2 — launch the GUI empty and pick files inside

rgbd-pd-viewer

The viewer opens with an empty scene. Use the sidebar:

• Load Primary RGBD… — two-step file dialog: pick depth first, then RGB.
• Camera Intrinsics — edit fx / fy / cx / cy and click Apply Intrinsics; the current point cloud (and any extra sources) is re-projected immediately.
• + Add Source (depth + RGB) — overlay additional point clouds on the same scene.

You can also pre-fill intrinsics so you don't have to type them in the UI:

rgbd-pd-viewer --intrinsics 640,640,320,240

Directory of frames (multi-frame sequence)

Pairs are matched by stripping --depth-suffix from each depth filename and looking for <stem><rgb-suffix> in --rgb-dir.

rgbd-pd-viewer \
  --depth-dir    /path/to/depth_npy \
  --rgb-dir      /path/to/color_jpg \
  --intrinsics   640,640,320,240 \
  --depth-suffix _480_640.npy \
  --rgb-suffix   .jpg

Pre-loading a previous calibration

rgbd-pd-viewer color.png depth.npy --t-wc-init T_wc_real.json

The JSON is expected to contain either a flat robot_base_in_camera_xyz field or a 4 × 4 T_cw matrix; both forms are written by the viewer's own "Save T_wc to JSON…" button.

---

All CLI flags

positional (single-frame mode)
  RGB                        path to RGB image  (.jpg / .png / .bmp / .tiff)
  DEPTH                      path to depth file (.npy / .npz)
  INTRINSICS                 fx,fy,cx,cy (4 floats, comma-separated, pixels)
                             default: 640,640,320,240

flags
  --intrinsics fx,fy,cx,cy   same format as the positional, useful with
                             directory mode or when launching an empty UI

directory mode (multi-frame)
  --depth-dir DIR            directory of depth files
  --rgb-dir DIR              directory of RGB images
  --depth-suffix STR         match depth files by suffix (default _480_640.npy)
  --rgb-suffix STR           match RGB files by suffix (default .jpg)

rendering
  --depth-scale FLOAT        multiplier on raw depth (default 1.0;
                             RealSense uint16 mm → use 0.001)
  --max-depth FLOAT          clip Z >= this in metres (default 10.0)
  --max-points INT           uniform-stride subsample cap (default: no cap)

optional calibration
  --t-wc-init PATH           JSON from a previous "Save T_wc" — pre-populates
                             the Robot Base Marker on launch

---

On-screen controls

Action	Input
Rotate view	Left-drag, or Azimuth / Elevation sliders
Zoom	Scroll wheel, or Distance slider
Pan	Right-drag
Prev / Next frame	< Prev / Next > buttons
Jump to specific frame	Type N, click Go
Toggle world axes	Grid: OFF / ON
Toggle camera frustum	Camera Marker: ON / OFF
Pick a 3-D point	Shift + Left-click
Drop sphere at the last pick	Place sphere at last pick
Nudge sphere along world X/Y/Z	X- / X+ / Y- / Y+ / Z- / Z+
Rotate world frame Rx / Ry / Rz	Rx- / Rx+ etc. (step in degrees)
Save calibration	Save T_wc to JSON…
Add another (depth, RGB) source	+ Add Source (depth + RGB)
Affine depth correction	edit A, B, click Apply Scale

---

Library use

The functions and the GUI class are also importable:

from rgbd_pd_viewer import (
    PointCloudApp,
    build_o3d_pointcloud,
    crop_and_resize,
    load_frame,
    get_sorted_frame_pairs,
)

# 1) Just back-project a depth+RGB pair to an Open3D PointCloud:
depth, rgb = load_frame("depth.npy", "color.png")
pcd = build_o3d_pointcloud(
    depth, rgb,
    fx=640, fy=640, cx=320, cy=240,
    depth_scale=1.0, max_depth=10.0,
)

# 2) Or launch the full GUI on a list of pairs:
pairs = get_sorted_frame_pairs("depth_dir", "rgb_dir",
                               depth_suffix="_480_640.npy",
                               rgb_suffix=".jpg")
app = PointCloudApp(pairs, fx=640, fy=640, cx=320, cy=240)
app.run()

Supported depth formats

Suffix	Loaded as
.npy	float32 array; if 3-D, channel 0 is taken
.npz	key "depth" if present, else npz.files[0]
.png/.jpg/…	RGB image (uint8, 3 channels)

Depth values are interpreted in metres after multiplication by --depth-scale. For RealSense raw uint16 saved as .npy, you typically want --depth-scale 0.001.

Robot Base Marker — JSON schema

Saving writes:

{
  "robot_base_in_camera_xyz": [x, y, z],
  "T_cw": [[...], [...], [...], [...]],
  "T_wc": [[...], [...], [...], [...]],
  "R_cw": [[...], [...], [...]],
  "rotation": "identity (only translation set) | manually adjusted via Rx/Ry/Rz buttons",
  "convention": "World origin = robot base. T_cw maps world->camera; T_cw[:3,:3] = R_cw (cols = world axes in camera frame); T_cw[:3,3] = robot-base position in camera frame. T_wc = inv(T_cw).",
  "source": "manual alignment via rgbd_pd_viewer Robot Base Marker"
}

---

Building & publishing

pip install build twine
python -m build                 # creates dist/*.whl and dist/*.tar.gz
twine check dist/*
twine upload dist/*             # or: twine upload -r testpypi dist/*

---

License

MIT © Yuzhen Chen