chirppy

Python library for reading and processing CHIRP datasets. It also has a PyTorch Dataloader to generate a CHIRP dataset ready to be used with PyTorch.

CHIRP (Cosys-Lab Hybrid In-air sonar Recording Package) is a HDF5-based dataset format designed for in-air active acoustic sensing. It provides a unified, easy-to-use standard for multi-channel ultrasonic array data, tightly fused with other sensors like LiDAR, Radar, Camera, IMU, GNSS, and Event Camera modalities for robotics research.

CHIRP is acoustic-focused: the acoustic sensor is the reference sensor, defining the grid of "reference captures" every other modality is synchronized to.

Installation

Dependencies

pip install -r requirements.txt

Quick Start

Basic Dataset Loading

from chirppy import CHIRPDataset

# Load dataset
dataset = CHIRPDataset('path/to/dataset.h5')

# Access sensors by property (if using standard formats)
camera = dataset.get_camera(0)      
lidar = dataset.get_lidar(0)       
acoustic = dataset.get_acoustic_sensor(0) 
imu = dataset.get_imu(0)           
gnss = dataset.get_gnss(0)
radar = dataset.get_radar(0)
event_camera = dataset.get_event_camera(0)

# Or access directly if sensor exists
camera = dataset.sensors['camera_00']

# Get specific capture. Indexing a property/array like this indexes whatever's actually stored -
# by default camera/lidar/radar/gnss are full_capture, so capture_idx=0 is just each sensor's own
# first raw sample, NOT necessarily acoustic reference capture 0. Use at_reference_capture(idx)
# when you want the sample nearest acoustic reference capture idx.
capture_idx = 0
raw_image = camera.images[capture_idx]     # raw: camera's own first sample
image = camera.at_reference_capture(capture_idx)['image']  # synced: nearest sample to reference capture 0
points = lidar.points[capture_idx]         # (max_points, 4) float32 [x, y, z, intensity]
signals = acoustic.signals[capture_idx]    # (samples, channels) float32
accel = imu.acceleration[capture_idx]      # (3,) float32
lat, lon = gnss.latitude[capture_idx], gnss.longitude[capture_idx]  # WGS84 decimal degrees
heading = gnss.heading[capture_idx]        # degrees, dual-antenna attitude (NaN without a solution)
radar_points = radar.get_all_points(capture_idx)  # (N, 7) float32 [x, y, z, doppler_velocity, rcs, motion_status, mode]

# Event camera captures work like any other full_capture sensor, each capture holds a variable-length list of events instead of a fixed-shape image:
capture = event_camera.at_reference_capture(capture_idx)  # nearest capture's events, synced
print(capture['event_count'], 'events:', capture['x'], capture['y'], capture['p'], capture['t'])
# Or query by time window directly, independent of capture boundaries:
t = acoustic.get_timestamps()[capture_idx]
events = event_camera.events_in_window(t - 0.05, t + 0.05)  # dict of x, y, p, t arrays

# Get calibration data
K = camera.get_intrinsics()              # (3, 3) intrinsic matrix
D = camera.get_distortion()              # (5,) distortion coefficients
to_origin = lidar.get_transform_to_origin()    # (3, 4) sensor→world
to_cam = lidar.get_transform_to_camera()       # (3, 4) sensor→cam_00
mic_positions = acoustic.get_receiver_relative_translations()  # (N_mics, 3)

# Get all sensors of a type
all_cameras = dataset.get_sensors_by_type('camera')
all_lidars = dataset.get_sensors_by_type('lidar')
all_radars = dataset.get_sensors_by_type('radar')
all_event_cameras = dataset.get_sensors_by_type('event_camera')

# Iterate through acoustic-synced captures (use at_reference_capture, not camera[i] - len(camera)
# is the native sample count when sync_mode == 'full_capture', not the reference-capture count)
for i in range(camera.num_reference_captures):
    capture_data = camera.at_reference_capture(i)
    image = capture_data['image']
    timestamp = capture_data['timestamp']
    # Process capture...

PyTorch DataLoader

from chirppy import CHIRPDatasetPyTorch
from torch.utils.data import DataLoader

# Create dataset with selected sensors
dataset = CHIRPDatasetPyTorch(
    'path/to/dataset.h5',
    sensors=['camera_00', 'lidar_00', 'acoustic_sensor_00', 'imu_00']
)

# Create dataloader with batching
dataloader = DataLoader(
    dataset,
    batch_size=4,
    shuffle=True,
    num_workers=2
)

# Iterate through batches
for batch in dataloader:
    images = batch['camera_00']['image']             # (B, H, W, C)
    points = batch['lidar_00']['points']             # (B, N, 4)
    masks = batch['lidar_00']['mask']                # (B, N) bool - valid points
    signals = batch['acoustic_sensor_00']['signals'] # (B, samples, channels)
    accel = batch['imu_00']['acceleration']          # (B, 3)
    
    # Process batch...

Generic HDF5 Access

# Read arbitrary datasets by path
custom_data = dataset.read_dataset('/sensors/custom_sensor/data/values')
metadata = dataset.read_attribute('/sensors/camera_00', 'sensor_type')

# List available datasets and groups
datasets = dataset.list_datasets('/sensors/camera_00/data')
groups = dataset.list_groups('/sensors')

Flexible Sensor Loading

# Access sensors by name (even if not standard)
custom_sensor = dataset.get_sensor('custom_sensor', create_if_missing=True)
if custom_sensor:
    # Access data through generic sensor interface
    timestamps = custom_sensor.timestamps

Examples

See the examples/ directory for complete usage examples:

• basic_example.py - Dataset loading, sensor access, visualization, coordinate frame plotting

• gnss_example.py - GNSS track visualization (lon/lat scatter colored by fix mode, dual-antenna heading arrows)

• radar_example.py - Radar point cloud visualization (3D scatter colored by Doppler, RCS, or motion status)

• event_camera_example.py - Event camera windowed queries and an accumulated-frame visualization

• pytorch_example.py - PyTorch DataLoader with batching and multi-worker loading

• odometry_example.py - Robot EKF vs. wheel odometry trajectories from the recording start Analysis tools (applications built on chirppy, not part of the library; Python only) are in tools/:

• projection.py - project_points(K, points, D, model) (pinhole, plumb-bob or fisheye) and project_to_camera(ds, points, source, camera, ...), which projects points of any sensor into a calibrated lens camera using the stored transforms and intrinsics

• verify_sync_motion.py - Checks the sensors' timestamps against each other by comparing how each one sees the robot move (the odometry's pose differentiated into speed and yaw rate - its reported velocity fields are an averaged, late copy - the IMU gyroscope, radar Doppler, LiDAR scan rotation, camera/GoPro frame change, event rate) and cross-correlating them to a per-sensor time offset; --against imu tells a late odometry from early sensors

• verify_transforms.py - Cross-checks the sensor transforms by projecting between sensors (LiDAR vs. depth image, radar vs. LiDAR, overlays on every camera, ground plane, IMU gravity, sync report)

Coordinate Systems

Every sensor stores its data in one data frame, and to_origin maps that frame to the origin:

1. Robot axes (origin frame, LiDAR, radar, IMU, GNSS, acoustic, odometry)

• Convention: Right-handed, X-forward, Y-left, Z-up
• Origin: the acoustic array center

2. Optical frame (every lens camera: RGB, thermal, event, video)

• Convention: Right-handed, Z-forward, X-right, Y-down (KITTI / OpenCV), so K applies to a point in this frame directly
• Origin: the camera's optical center

to_cam_00 is inv(camera_00 to_origin) @ sensor to_origin: the sensor's data in the reference camera's optical frame (the reference camera's own to_cam_00 is the identity).

Transform example:

# Any pair of sensors: 4x4 T with p_target = T @ p_source, via the origin frame
T = dataset.get_transform('lidar_00', 'thermal_camera_00')
points_in_thermal_frame = dataset.transform_points(lidar_points, 'lidar_00', 'thermal_camera_00')

# projecting points into a camera is application code: see tools/projection.py

# the same by hand with the stored matrices
P_cam = lidar.get_transform_to_camera() @ [x, y, z, 1]
u, v = (camera.get_intrinsics() @ P_cam[:3])[:2] / P_cam[2]

To check that all transforms are right, run tools/verify_transforms.py dataset.h5: it cross-checks sensors against each other (LiDAR range vs. the RealSense depth image, radar vs. LiDAR, ground plane, IMU gravity, sync) and writes LiDAR/radar overlays on every camera.

Data Types

• Images: uint8 RGB format (N, H, W, 3)
• LiDAR point clouds: float32 [x, y, z, intensity], NaN-padded to fixed size
• Radar point clouds: float32 [x, y, z, doppler_velocity, rcs, motion_status, mode], NaN-padded to fixed size
• Event camera data: x/y uint16, p uint8 (1=ON/0=OFF), t float64 Unix epoch seconds - one flat, time-ordered array per field across all captures, with capture_event_start/capture_event_count locating each capture's slice
• Acoustic signals: float32 (N, samples, channels) format
• IMU data: float32 [ax, ay, az] or [wx, wy, wz]
• GNSS data: float64 latitude/longitude/height, velocity_neu, heading/pitch/roll (NaN without a dual-antenna solution), course_over_ground (NaN when stationary/no fix), position/velocity/attitude covariances uint8 fix_mode/num_satellites
• Timestamps: float64 Unix epoch seconds
• Transforms: float64 (4, 4) homogeneous transformation matrices

API Reference

CHIRPDataset

Main dataset class for reading CHIRP HDF5 files.

Attributes:

• format - Format name (should be "chirp")
• version - Format version string
• n_reference_captures - Number of acoustic reference captures
• reference_sensor - Name of reference sensor for synchronization
• sensors - Dictionary of all loaded sensor objects
• h5file - Underlying h5py.File object

Methods:

Sensor Access:

• get_sensor(name, create_if_missing=True) - Get sensor by name, optionally loading if not already loaded
• get_sensors_by_type(sensor_type) - Get all sensors of specific type (e.g., 'camera', 'lidar')
• get_camera(idx=0) - Get camera sensor by index (e.g., camera_00)
• get_lidar(idx=0) - Get LiDAR sensor by index
• get_acoustic_sensor(idx=0) - Get acoustic receiver array by index
• get_acoustic_transmitter(idx=0) - Get acoustic transmitter by index
• get_imu(idx=0) - Get IMU sensor by index
• get_gnss(idx=0) - Get GNSS sensor by index
• get_radar(idx=0) - Get radar sensor by index
• get_event_camera(idx=0) - Get event camera sensor by index
• get_video(idx=0) - Get video sensor by index (frames live in an MP4 beside the .h5)
• get_thermal_camera(idx=0) - Get thermal camera sensor by index (raw radiometric frames)
• get_odometry(idx=0) - Get odometry sensor by index (0 = EKF, 1 = wheel)

Reference-capture access (the acoustic sensor defines the grid):

• dataset[i] / get_reference_capture(i, sensors=None) - every sensor's sample nearest reference capture i, keyed by sensor name
• reference_timestamps - Unix-epoch times of the reference captures

Frames and projection:

• get_transform(source, target) - 4x4 T with p_target = T @ p_source between any two sensors (names or objects) or the frames 'origin', 'base_link' and 'ground' (the last two need format v2.1), in each sensor's data frame (see Coordinate Systems)
• transform_points(points, source, target) - the same for (N, >=3) points; extra columns (intensity, doppler, ...) are kept
• origin_height_above_ground, base_link_height_above_ground - heights in meters (nominal), None for v2.0 datasets without a ground frame

Generic HDF5 Access:

• read_dataset(path, selection=...) - Read arbitrary HDF5 dataset by path, optionally only a slice (e.g. slice(0, 10))
• read_attribute(path, attr_name) - Read HDF5 attribute
• list_datasets(group_path) - List all datasets under path
• list_groups(group_path) - List all groups under path

Special Methods:

• __len__() - Returns number of reference captures
• __enter__(), __exit__() - Context manager support
• close() - Close HDF5 file

Sensor Classes

Every sensor also has (see BaseSensor):

• indices_in_window(t_start, t_end) / get_window(t_start, t_end) - native captures with timestamp in [t_start, t_end); ImuSensor.get_window returns stacked arrays, EventCameraSensor.events_in_window returns events
• points_to_origin(points) - express (N, >=3) points from the sensor frame in the origin frame (T_to_origin @ [x, y, z, 1]), extra columns kept
• can_align_to_reference - False for a full_capture sensor without sync_to_reference_index
• Unknown sensor types load as GenericSensor (timestamps, transforms, raw datasets)
• PyTorch: CHIRPDatasetPyTorch works with DataLoader(num_workers>0); pass collate_fn=chirp_collate to batch variable-length data such as event-camera captures

All sensors inherit from BaseSensor and provide common interface:

BaseSensor (Abstract Base):

• sensor_type (property) - Type string (e.g., 'camera', 'lidar')
• reference_frame (property) - Reference coordinate frame
• timestamps (property) - Capture timestamps array
• calibration (property) - Calibration data namespace
• get_timestamps() - Get timestamps array
• get_transform_to_origin() - Get sensor→world transform (3, 4)
• get_transform_to_camera() - Get sensor→cam_00 transform (3, 4)
• __len__() - Number of captures (native sample count if is_full_capture, else n_reference_captures)
• __getitem__(idx) - Get capture data dictionary (indexes the array as actually stored - see sync_mode)
• sync_mode (property) - 'full_capture' (default) or 'reference_capture'; see README.md's "Full-Capture vs. Reference-Capture Storage"
• is_full_capture (property) - Shorthand for sync_mode == 'full_capture'
• num_reference_captures (property) - Number of acoustic reference captures this sensor aligns to (always n_reference_captures, regardless of sync_mode)
• at_reference_capture(capture_idx, **kwargs) - Get the sample nearest acoustic reference capture capture_idx, regardless of sync_mode - use this instead of sensor[idx] whenever you specifically want the acoustic-synced view (**kwargs forwarded to __getitem__, e.g. keep_padding for LiDAR/radar)

Lens cameras (CameraSensor, ThermalCameraSensor, EventCameraSensor, VideoSensor) share get_intrinsics() (K), get_distortion() (D), get_image_size(), distortion_model ('plumb_bob' or 'equidistant' fisheye), and is_calibrated. Intrinsics appear once a calibration file is set for that camera in transforms/sensor_objects_description.json (cam_file); until then is_calibrated is False.

CameraSensor:

• images (property) - Image array (N, H, W, 3)
• has_depth (property) - Whether this recording captured depth (--camera-depth) alongside color - most don't, it's off by default
• depth (property) - Depth array (N, H, W), uint16 millimeters, aligned to images, or None if has_depth is False
• get_depth_meters(idx) - Depth frame at index converted to float32 meters, or None if has_depth is False
• __getitem__(idx)'s 'depth' key is None when has_depth is False, otherwise the same (H, W) uint16-millimeter array as depth[idx]
• calibration.K - Intrinsic matrix (3, 3)
• calibration.D - Distortion coefficients (5,)
• calibration.image_size - Image dimensions (2,)
• get_intrinsics() - Get K matrix
• get_distortion() - Get distortion coefficients
• get_image_size() - Get [height, width]
• get_factory_calibration() - The calibration the camera itself reported when this was recorded, as a dict of arrays, or None for older recordings: color_K/color_D (intrinsics of the stored images, adjusted for the 180° mounting rotation), color_K_sensor/color_D_sensor/depth_K_sensor/depth_D_sensor (as the SDK reports them), 4×4 depth_to_color/accel_to_depth/gyro_to_depth (p_dst = T @ p_src, meters), and calibration_table (raw bytes that can be written back to a camera). Separate from calibration.K/D, which are your own calibration
• get_device_info() - Full snapshot taken at the start of the recording (device name, serial, firmware, stream settings, depth-sensor options such as visual preset and laser power, depth filter chain) as a dict, or None for older recordings
• device_serial (property) - Serial number of the camera that recorded this, if known

LidarSensor:

• points (property) - Point cloud array (N, max_points, 4)
• point_counts (property) - Valid points per capture (N,)
• max_points (property) - Maximum points capacity
• get_all_points(idx, remove_padding=True) - Get points for capture with optional padding removal
• __getitem__(idx, keep_padding=False) - Get capture with control over NaN padding

RadarSensor:

• points (property) - Point cloud array (N, max_points, 7) [x, y, z, doppler_velocity, rcs, motion_status, mode]
• point_counts (property) - Valid points per capture (N,)
• max_points (property) - Maximum points capacity
• get_all_points(idx, remove_padding=True) - Get points for capture with optional padding removal
• __getitem__(idx, keep_padding=False) - Get capture with control over NaN padding
• motion_status values: -1 moving towards, 0 standing, 1 moving away - classified per-point against a per-capture ego-velocity estimate (see the altos-v2-py driver, which computes this at record time)
• mode: which of the sensor's two interleaved range-profile configs (short/long range) this point came from - each capture already combines the nearest-in-time frame of both (see convert_raw_to_chirp's _pair_radar_frames), so points from different modes can differ in resolution/unambiguous velocity even within one capture

EventCameraSensor: a capture holds a variable-length list of events, so __len__/__getitem__/at_reference_capture/sync_mode all work exactly like every other sensor.

• __len__() - Number of captures
• __getitem__(idx) - Get capture at index: {x, y, p, t, event_count, timestamp} (x/y/p/t are that capture's events; timestamp is the capture's own host timestamp, not t[0]/t[-1].
• at_reference_capture(capture_idx) - Get the events from the capture nearest acoustic reference capture capture_idx, regardless of sync_mode.
• x, y, p (properties) - Direct access to the flat, all-captures-concatenated event datasets (N_events,)
• capture_event_start, capture_event_count (properties) - Direct access to the per-capture index arrays (N_captures,)
• image_width, image_height (properties) - Sensor resolution in pixels
• events_in_window(t_start, t_end) - Get every event with t in [t_start, t_end), independent of capture boundaries, useful when you want a time window that doesn't line up with a single capture. Note: each capture is timestamped independently against the host clock rather than one shared anchor for the whole recording

ThermalCameraSensor: raw radiometric frames from a FLIR A70. __getitem__(i) returns temperature_c (float32, degC), the stored raw uint16 (temperature_K = raw * kelvin_per_count), timestamp, device_timestamp_ns (the camera's own free-running clock in nanoseconds - not Unix time, and -1 when the camera delivered a corrupt reading), repeated and frozen. Frames inside a recalibration (NUC) freeze are kept and flagged; filter with the frozen mask. get_temperature_c(i) returns temperatures only. colorize(i, palette, temperature_range) produces a false-color image with the iron, inferno or gray palette, with no extra dependency. Pass a fixed temperature_range to keep colors comparable between frames; the default stretches each frame on its own. measurement_parameters gives the emissivity etc. the camera used.

OdometrySensor: the robot's own odometry, one sensor per stream: odometry_00 = EKF, odometry_01 = wheel (source says which). __getitem__(i) returns base_link's position (3,) and orientation [x, y, z, w] in the odom frame, yaw (radians), linear_velocity/angular_velocity in base_link, pose_covariance/twist_covariance (6, 6), timestamp and ros_timestamp. Right-handed X forward, Y left, Z up. Timing: on the Husky, linear_velocity and angular_velocity are averaged over roughly 0.8-1 s and stamped at the end of the window, so they trail the robot's real motion by about 0.4 s (measured against the LiDAR IMU), whereas position and orientation are in sync with the other sensors to within ~50 ms. For anything that depends on timing, differentiate the poses (poses()) instead of using the reported velocities. get_pose_matrix(i) and poses() give 4x4 poses; relative_poses(start=0) makes the trajectory start at the identity (the odom frame starts where the driver started, not the recording). path_length() is the distance driven. Example: examples/odometry_example.py.

VideoSensor: the payload is an MP4 sibling of the .h5, not data inside it, so this class stores no frames, it only holds configuration parameters and time-sync information. __getitem__(i) returns {frame_index, timestamp, pts_s, video_path} and deliberately does not decode, so iterating is cheap; get_image(i) decodes a single frame by shelling out to ffmpeg.

frame_for_reference_capture(i) returns (frame_index, timestamp, valid, dt_s), and at_reference_capture(i) adds sync_valid/sync_dt_s/reference_timestamp to the returned dict.

set_manual_offset(seconds, persist=False) corrects the alignment without re-running a conversion: the offset is applied at read time, and the mapping is re-derived from the stored reference_timestamps. Positive means the video actually started later than estimated - so a fixed wall-clock capture then lands on an earlier video frame.

video = dataset.get_video()
frame_idx, t, valid, dt = video.frame_for_reference_capture(137)
if valid:
    image = video.get_image(frame_idx)

AcousticSensor (Microphone Array):

• signals (property) - Audio signals (N, samples, channels)
• num_channels (property) - Number of microphones
• calibration.receiver_relative_translations - Microphone positions (N_mics, 3)
• get_receiver_positions() - Alias for receiver_relative_translations
• get_receiver_relative_translations() - Get microphone 3D positions

AcousticTransmitter:

• signals (property) - Emitted signal waveform (N_samples,)
• get_emitted_signal() - Get emitted chirp/sweep signal
• __getitem__(idx) - Get emission timestamp for capture

ImuSensor:

• acceleration (property) - Linear acceleration (N, 3)
• angular_velocity (property) - Angular velocity (N, 3)

GnssSensor: curated to the fields useful for navigation/SLAM/robotics fusion - GNSS-diagnostic-only fields (DOP, protection levels, MSL/geoid height, GNSS-derived attitude rates, scalar accuracy, error flags) aren't exposed here; the raw per-epoch JSON in record_raw's gnss/ folder still has the full field set.

• latitude / longitude (property) - WGS84 position (N,) decimal degrees
• height (property) - Height above the WGS84 ellipsoid (N,) meters
• velocity_neu (property) - NEU velocity (N, 3) m/s
• heading / pitch / roll (property) - Dual-antenna attitude (N,) degrees, NaN without a solution
• course_over_ground (property) - Movement-derived heading (N,) degrees, NaN when stationary/no fix; None if absent (datasets converted before this field was added)
• position_covariance / velocity_covariance / attitude_covariance (property) - Covariance arrays (N, 10) / (N, 10) / (N, 6)
• fix_mode (property) - Fix quality (N,): 0=No Fix, 1=Stand-Alone, 2=DGPS, 3=Fixed, 4=RTK Fixed, 5=RTK Float, 6=SBAS, 7=MB RTK Fixed, 8=MB RTK Float, 9=PPP
• fix_mode_string (property) - Human-readable fix label derived from fix_mode
• num_satellites (property) - Satellites used in fix (N,)
• gnss_time (property) - Unix epoch UTC (N,), from GPS time-of-week/week number

CHIRPDatasetPyTorch

PyTorch-compatible dataset with fixed-size tensor outputs and validity masks. Always yields the acoustic-synced view (n_reference_captures samples) regardless of whether any included sensor has sync_mode == 'full_capture' - it resolves each sample internally via at_reference_capture(), so training loops never need to deal with full-capture/reference-capture mismatches.

Constructor:

CHIRPDatasetPyTorch(
    h5_path: str,
    sensors: Optional[List[str]] = None,
    transform: Optional[Callable] = None,
    target_transform: Optional[Callable] = None
)

Parameters:

• h5_path - Path to HDF5 file
• sensors - List of sensor names to load (None = all sensors)
• transform - Optional transform to apply to data
• target_transform - Optional transform for targets/labels

Methods:

• __len__() - Number of samples
• __getitem__(idx) - Get multi-modal sample as PyTorch tensors
• get_sensor_list() - Get list of loaded sensor names
• get_base_dataset() - Get underlying CHIRPDataset object

Returns:

• Dictionary with sensor data as PyTorch tensors
• Note that LiDAR and radar include an additional field mask (bool tensor) for valid points (masks NaN padding)

License

This project is released under the CC-BY-NC-SA-4.0 License. Please review the License file for more details.