pyrobovision 0.5.0

Advanced autonomous driving perception and vision-language foundation models for robotics

PyRoboVision

Advanced autonomous driving perception and vision-language foundation models for robotics. Built on top of PyRoboFrames.

Focus: Advanced perception pipelines and multimodal understanding that consume data loaded by PyRoboFrames.

---

What's Inside

Autonomous Driving (v0.5)

• Cylindrical panoramic stitching — 360° multi-camera fusion (Waymo, nuScenes)
• Advanced blending — Laplacian pyramid + graph-cut seams
• Bird's-eye-view (BEV) — 3D projection for autonomous perception
• GPU acceleration — CuPy (NVIDIA), MLX (Apple Silicon), NumPy (CPU)
• Sensor fusion — Lidar/Radar + occupancy grid mapping
• Dataset loaders — Waymo TFRecord, nuScenes JSON, KITTI stereo

Foundation Models (Phase 7)

• SAM3 segmentation — Instance segmentation + temporal tracking
• CLIP embeddings — Scene understanding, text-image similarity
• Grounding DINO — Open-vocabulary object detection
• Multi-modal fusion — Unified detection + segmentation + classification

---

Installation

# Requires PyRoboFrames v1.0+
pip install pyroboframes pyroboframes-vision

# From source
git clone https://github.com/Mullassery/PyRoboVision.git
cd PyRoboVision
pip install -e .

---

Quick Start

Autonomous Driving: 360° Panoramic Perception

from pyrobovision.automotive import (
    CylindricalStitcher,
    get_waymo_layout,
)

# Stitch 5 cameras into 360° panorama
layout = get_waymo_layout()
stitcher = CylindricalStitcher(layout, blend_method="laplacian")

frames = {
    "FRONT": ...,
    "FRONT_LEFT": ...,
    # ... other cameras
}

panorama = stitcher.stitch(frames)  # [1, H, W, 3] seamless 360°

Foundation Models: Multi-Modal Scene Understanding

from pyrobovision.foundation_models import MultiModalFusion

fusion = MultiModalFusion(
    detection_prompt="car . pedestrian . cyclist",
    device="mlx",  # or "cuda"
)

scene = fusion.understand(frame)
for obj in scene.objects:
    print(f"{obj.object_class}: {obj.semantic_label}")

---

Architecture

Dependency Graph

PyRoboVision/
├── automotive/          # v0.5 AV perception
│   ├── stitching.py
│   ├── blending.py
│   ├── bev.py
│   ├── perception_3d.py
│   ├── tfrecord_utils.py
│   ├── nuscenes_utils.py
│   └── datasets.py
│
└── foundation_models/   # Phase 7
    ├── sam3_segmentation.py
    ├── clip_embeddings.py
    ├── grounding_dino.py
    └── multimodal_fusion.py

↓ Depends on PyRoboFrames v1.0+ (dataloader)
PyRoboFrames/
├── RoboFrameDataset      # Load LeRobot
├── ProprioceptiveLoader  # Load state/action
├── DataLoader            # Device selection
└── [video decode, sensor fusion, etc.]

Key design: PyRoboVision is a consumer library, not a foundation. It uses PyRoboFrames to load data, then applies perception algorithms.

---

Features

Phase	Feature	Status	Tests
1	Cylindrical panoramic projection	✅	10
2	Laplacian pyramid blending	✅	5
3	Bird's-eye-view (BEV) projection	✅	5
4a	GPU acceleration (CuPy/MLX/NumPy)	✅	6
4b	Optical flow seam tracking	✅	10
5	Waymo/nuScenes/KITTI loaders	✅	9
6	Lidar/Radar fusion + Occupancy grids	✅	18
7a	SAM3 temporal segmentation	✅	18
7b	CLIP scene embeddings	✅	25
7c	Grounding DINO detection	✅	26
7d	Multi-modal fusion	✅	17

Total: 149 tests, all passing

---

Use Cases

Autonomous Driving

• Waymo perception pipeline (panoramic stitching + 3D fusion)
• nuScenes multi-camera understanding
• Real-time BEV mapping

Mobile Manipulation

• Egocentric robot perception (360° view from mobile base)
• Scene understanding for pick-and-place

Robotdog Navigation

• Panoramic localization (where am I in the scene?)
• Terrain classification from multi-camera fusion

---

Related Projects

• PyRoboFrames — Fast ML dataloader for robot learning (core dependency)
• LeRobot — HuggingFace robotics datasets
• Segment Anything 3 (SAM3) — Instance segmentation
• CLIP — Vision-language models
• Grounding DINO — Open-vocabulary detection

---

License

MIT (same as PyRoboFrames)

---

Contributing

Contributions welcome. Please open issues and PRs on GitHub.

For architectural decisions, see ARCHITECTURE.md.

---

Citation

@software{mullassery2025pyrobovision,
  title={PyRoboVision: Advanced perception and vision-language models for robotics},
  author={Mullassery, Georgi},
  url={https://github.com/Mullassery/PyRoboVision},
  year={2025}
}