phys-ai-dataprofiler

Profile physical-AI and lab datasets on the Hugging Face Hub without downloading them, and check them against what your model needs.

It reads only the bytes that hold metadata (headers, indexes, footers) through HTTP range requests, usually well under 1% of each file, and reports streams, message counts, rates, jitter, gaps, episodes, NaN counts and problems such as truncated or unindexed recordings. A requirements file turns that into a pass / fail / unknown verdict, and compare ranks candidate datasets side by side.

Measured on public Hub data: a 5.8 GB ROS 1 bag profiled from 533 KB, a 2.6 GB MAT v7.3 file from 808 KB.

Install

pip install phys-ai-dataprofiler
pip install "phys-ai-dataprofiler[mat73]"      # MAT v7.3 (HDF5) files; needs Python 3.13 or older for h5py

Python 3.10 or newer. Public datasets need no account; for gated or private ones set HF_TOKEN (or run hf auth login). The token is only sent to huggingface.co, never to the CDN hosts that serve file bytes.

Quick start

phys-ai-dataprofiler org/dataset --dry-run              # what files are there, and how big
phys-ai-dataprofiler org/dataset --max-files 10         # profile a sample -> reports/<repo>__<time>.md/.json

phys-ai-dataprofiler requirements list                  # built-in presets
phys-ai-dataprofiler requirements show teleop-imitation > req.yaml   # start your own, edit the numbers
phys-ai-dataprofiler check org/dataset -r req.yaml      # verdict; exit 0 pass, 1 fail, 4 unknown
phys-ai-dataprofiler compare org/a org/b -r req.yaml    # rank candidates against the same requirements

Nothing is downloaded unless you ask for it (--mode download, or --full-read-max-mb for small files). Nothing is ever written to the Hub.

Requirements and verdicts

Say what your model needs in a YAML file, and the profiler answers pass, fail or unknown for each requirement, with evidence and, for unknowns, how to settle them. Checks are deterministic rules, with no model call. Full reference: docs/REQUIREMENTS.md.

phys-ai-dataprofiler check org/dataset -r preset:teleop-imitation        # exit 0 pass, 1 fail, 4 unknown
phys-ai-dataprofiler compare org/a org/b -r my_requirements.yaml          # rank candidate datasets
phys-ai-dataprofiler check --report reports/run.json -r my_requirements.yaml   # re-check offline

streams:
  - name: camera
    match: {modality: video}
    min_rate_hz: 25
  - name: joint state
    match: {topic: ["/joint_states", "observation.state"]}
    min_rate_hz: 50
    max_gap_ms: 100
dataset:
  min_total_hours: 20
  min_episodes: 500

A number that was sampled and lies within 10% of the threshold, a NaN count the format cannot provide, or a file that failed to profile makes the result unknown, never a guessed pass.

ROS bag support

.bag (ROS 1, format V2.0) and .db3 (ROS 2, SQLite storage) files are profiled from byte ranges, like MCAP. ROS 2 bags in MCAP storage are plain .mcap files and use the MCAP path. Code: phys_ai_dataprofiler/remote/rosbag.py.

• ROS 1: the bag header points to the index at the end of the file (connections plus per-chunk counts and time ranges). Per-message timestamps come from the IndexData records after each chunk, 12 bytes per message, one request per chunk. Chunk payloads are never read, so compression (bz2, lz4) makes no difference. Over --msgidx-budget-mb, chunks are sampled; counts stay exact. Bags with no index (recorder killed) are walked record by record and reported unindexed_bag; truncated bags are reported truncated. Old V1.2 bags are identified but not parsed.
• ROS 2 SQLite: a small read-only SQLite reader descends the messages table B-tree. Each row's topic_id and timestamp sit at the start of its leaf cell, so message payloads and their overflow pages are never read. Counts and duration come from the metadata table, or from the metadata.yaml next to the .db3 when the database has none. Over budget, runs of leaf pages are sampled and the start and end times come from timestamp_idx. Without any metadata, sampled per-topic counts are estimates, reported as estimated_counts.

Timing statistics use the same code as MCAP. Tests compare every case against a full read through rosbags: python tests/test_rosbag_inspector.py (add --hub for two public bags).

Parquet support

Parquet keeps its schema, row groups and per-column statistics in a footer at the end of the file, so the profiler reads the footer first (typically under 100 KB, whatever the file size). It reports rows, row groups, every column's type, null count, min and max, compressed size and codec.

If a time column is present (timestamp, log_time, time, ...), it also reads that column, and episode_index when present (the LeRobot convention). Their byte ranges are known from the footer, so the cost is computed up front and capped by --column-budget-mb (default 16); beyond that, row groups are sampled and the report says so. From these it derives frame rate, jitter, gaps, episode count and lengths, and flags dropped frames, short episodes and non-monotonic time. Image, text and state columns are never read.

Units are inferred: float seconds (LeRobot), integer epoch s/ms/us/ns, and Arrow timestamp types. NaN/inf counts are not in Parquet statistics, so float columns are read too, capped by --nan-budget-mb (default 32); beyond that, row groups are sampled and the report says so. --nan-budget-mb 0 turns this off.

Repos with nothing to profile now say so and exit with code 3, listing the file types they do contain.

MCAP File Inspector

A tool to inspect MCAP (Message Capture) files and extract header, footer, channels, topics, and schema information.

Features

• Header Inspection: Extracts profile and library information
• Footer Inspection: Extracts summary start offset, summary offset start, and CRC
• Channel Discovery: Lists all channels with their topics, encoding, schema IDs, and metadata
• Topic Listing: Provides a sorted list of all unique topics
• Schema Information: Extracts schema details including name, encoding, and data
• Local and Remote Files: Supports both local file paths and remote URLs
• Fallback Support: Works with or without the mcap library installed

Installation

1. Create a virtual environment:

python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

2. Install dependencies:

pip install mcap

Usage

As a module

from phys_ai_dataprofiler.inspectors.mcap import McapFileInspector

# Inspect a local file
inspector = McapFileInspector("samples/sample.mcap")
result = inspector.inspect()

print(f"Topics: {result.topics}")
print(f"Channels: {len(result.channels)}")
print(f"Header Profile: {result.header.profile}")
print(f"Footer Summary Start: {result.footer.summary_start}")

As a command-line tool

# Using the basic inspect_file script
python -m phys_ai_dataprofiler.inspectors.mcap samples/sample.mcap

# Using the test script with formatted output
python tests/test_mcap_inspector.py samples/sample.mcap

Remote URLs

python tests/test_mcap_inspector.py https://example.com/data.mcap

Data Structures

McapInspectionResult

• url: File URL or path
• is_mcap: Boolean indicating if file is a valid MCAP
• magic: Magic bytes from file header
• size_bytes: File size in bytes
• path: Local path (if applicable)
• error: Error message (if any)
• header: HeaderInfo object
• footer: FooterInfo object
• channels: List of ChannelInfo objects
• schemas: List of SchemaInfo objects
• topics: Sorted list of topic strings
• mcap_available: Boolean indicating if mcap library is available

HeaderInfo

• profile: MCAP profile string
• library: Library that created the file

FooterInfo

• summary_start: Offset where summary section starts
• summary_offset_start: Offset where summary offsets start
• summary_crc: CRC checksum of summary section

ChannelInfo

• id: Channel ID
• topic: Topic name
• message_encoding: Message encoding (e.g., "json", "protobuf")
• schema_id: Associated schema ID
• metadata: Dictionary of channel metadata

SchemaInfo

• id: Schema ID
• name: Schema name
• encoding: Schema encoding type
• data: Raw schema data (bytes)

Example Output

Inspecting: samples/sample.mcap

URL: samples/sample.mcap
Is MCAP: True
Size: 1042 bytes
MCAP Library Available: True

=== Header ===
Profile: test
Library: sample_creator

=== Footer ===
Summary Start: 397
Summary Offset Start: 849
Summary CRC: 544820573

=== Topics (2) ===
  - /sensors/gps
  - /sensors/imu

=== Channels (2) ===
  Channel 1:
    Topic: /sensors/gps
    Encoding: json
    Schema ID: 1
    Metadata: {'description': 'GPS sensor data'}
  Channel 2:
    Topic: /sensors/imu
    Encoding: json
    Schema ID: 1
    Metadata: {'description': 'IMU sensor data'}

=== Schemas (1) ===
  Schema 1:
    Name: example.Point
    Encoding: jsonschema
    Data size: 84 bytes

Creating Test Files

Use the included create_sample_mcap.py script to create test MCAP files:

python scripts/create_sample_mcap.py output.mcap

Implementation Details

The inspector uses the official mcap library to parse MCAP files:

• make_reader() for high-level summary and header access
• StreamReader for low-level footer record extraction
• Supports both file handles and BytesIO for remote content

For remote files, the inspector first checks the magic bytes using HTTP Range requests, then downloads the full file for parsing if valid.

MATLAB File Inspector

A tool to inspect MATLAB .mat files and extract header information, variable metadata, and data types.

Features

• Header Inspection: Extracts description, version, and endianness information
• Variable Discovery: Lists all variables with their names, types, shapes, and sizes
• Type Detection: Identifies scalars, vectors, matrices, complex numbers, sparse matrices, structs, and cell arrays
• Format Support: Handles MAT-File v4, v5, v7, and v7.3 (HDF5-based) formats
• Local and Remote Files: Supports both local file paths and remote URLs
• Fallback Support: Works with limited functionality if scipy/h5py are not installed

Installation

1. Create a virtual environment (if not already created):

python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

2. Install dependencies:

pip install scipy numpy       # For v4/v5/v7 format support
pip install h5py              # For v7.3 (HDF5) format support (requires HDF5 library)

Usage

As a module

from phys_ai_dataprofiler.inspectors.mat import MatFileInspector

# Inspect a local file
inspector = MatFileInspector("data.mat")
result = inspector.inspect()

print(f"Format: {result.file_format}")
print(f"Variables: {result.variable_names}")
for var in result.variables:
    print(f"{var.name}: {var.dtype} {var.shape}")

As a command-line tool

# Using the basic matfile_inspector script
python -m phys_ai_dataprofiler.inspectors.mat data.mat

# Using the test script with formatted output
python tests/test_mat_inspector.py data.mat

Remote URLs

python tests/test_mat_inspector.py https://example.com/data.mat

Data Structures

MatInspectionResult

• url: File URL or path
• is_matfile: Boolean indicating if file is a valid MAT-File
• file_format: Format version ("v4", "v5", "v7", or "v7.3")
• size_bytes: File size in bytes
• path: Local path (if applicable)
• error: Error message (if any)
• header: MatFileHeader object
• variables: List of VariableInfo objects
• variable_names: Sorted list of variable names
• scipy_available: Boolean indicating if scipy is available
• h5py_available: Boolean indicating if h5py is available

MatFileHeader

• description: File description string (includes platform and creation time)
• version: Version number in hex format
• endian: Endianness ("little", "big", or "unknown")
• created: Creation timestamp (when available)

VariableInfo

• name: Variable name
• dtype: Data type string
• shape: Tuple of dimensions
• size_bytes: Size in bytes
• is_sparse: Boolean indicating sparse matrix
• is_complex: Boolean indicating complex numbers
• is_struct: Boolean indicating MATLAB struct
• is_cell: Boolean indicating cell array
• field_names: List of field names (for structs)

Example Output

Inspecting: samples/sample_v5.mat

URL: samples/sample_v5.mat
Is MAT-File: True
File Format: v5
Size: 1544 bytes
SciPy Available: True
h5py Available: False

=== Header ===
Description: MATLAB 5.0 MAT-file Platform: posix, Created on: Sun Oct 11 18:59:29 2026
Version: 0x0100
Endian: little

=== Variables (12) ===
  - col_vector
  - complex_data
  - double_data
  - float_data
  - int_array
  - logical_array
  - matrix_2d
  - matrix_3d
  - row_vector
  - scalar_value
  - text_string
  - uint_array

=== Variable Details ===

  scalar_value:
    Type: int64
    Shape: (1, 1)
    Size: 8 bytes

  complex_data:
    Type: complex128
    Shape: (1, 3)
    Size: 48 bytes
    Flags: complex

  matrix_3d:
    Type: float64
    Shape: (2, 3, 4)
    Size: 192 bytes

Supported MAT-File Formats

v4 Format

• Legacy format from MATLAB 4
• Limited data type support
• Requires scipy

v5 Format (Default)

• MATLAB 5 and later (default save format in older versions)
• Supports all standard MATLAB types
• Requires scipy

v7 Format

• Compressed v5 format
• Same structure as v5 but with compression
• Requires scipy

v7.3 Format (HDF5-based)

• MATLAB R2006b and later
• Based on HDF5 file format
• Required for variables larger than 2GB
• Requires h5py and HDF5 library

Creating Test Files

Use the included create_sample_mat.py script to create test MAT files:

python scripts/create_sample_mat.py samples/sample_v5.mat samples/sample_v7.mat

Implementation Details

The inspector uses two libraries depending on the MAT-File format:

• scipy.io.loadmat: For v4, v5, and v7 formats
• h5py: For v7.3 (HDF5-based) format

For remote files, the inspector first checks the file header using HTTP Range requests to determine the format, then downloads the full file for parsing if valid.

Comparison with MCAP Inspector

Both inspectors follow a similar architecture:

Hugging Face Connector

phys_ai_dataprofiler/connectors/huggingface.py (CLI: phys-ai-dataprofiler, or python -m phys_ai_dataprofiler) connects to the Hugging Face Hub with an access token, finds the MCAP / MAT files in a repo, inspects them and writes a JSON and a Markdown report. By default nothing is downloaded: files are read through HTTP Range requests and only the bytes holding metadata are fetched.

Remote mode gives the same streams, rates, gaps, findings and NaN counts as the full-read RichInspector (checked by tests/test_remote_rich.py). It also profiles truncated MCAPs, which the full read cannot open. Anything partial (budget reached, chunks sampled) is reported as a finding. Use --full-read-max-mb N to fetch files at or below that size whole instead (deleted afterwards unless --keep-downloads), or --mode download to go back to downloading everything.

Token

Resolved in this order: --token, then HF_TOKEN / HUGGINGFACE_HUB_TOKEN, then the login cached by hf auth login. Public repos work with no token. The token is only ever sent to huggingface.co, never to the CDN hosts that serve the file bytes.

Usage

# Inspect every .mcap/.mat file in a dataset repo without downloading
phys-ai-dataprofiler org/dataset --token hf_xxx

# See what would be inspected first
phys-ai-dataprofiler org/dataset --dry-run

# Limit the selection: by glob, by count, by size
phys-ai-dataprofiler org/dataset --include "episodes/*.mcap" --max-files 5

# Exact files
phys-ai-dataprofiler org/dataset --file logs/run1.mcap --file calib/cam.mat

# Larger NaN budget (default 32 MB of float data per file), or none
phys-ai-dataprofiler org/dataset --nan-budget-mb 256
phys-ai-dataprofiler org/dataset --nan-budget-mb 0

# Fetch small files whole instead of range-reading them
phys-ai-dataprofiler org/dataset --full-read-max-mb 50

# Classic download mode, and re-running on an earlier download
phys-ai-dataprofiler org/dataset --mode download --max-total-mb 2000
phys-ai-dataprofiler org/dataset --mode download --offline

# Model repos, specific revisions
phys-ai-dataprofiler org/model --repo-type model --revision v1.2

Tuning for remote mode: --workers (parallel range requests, default 16), --msgidx-budget-mb (cap on MCAP message-index bytes per file; beyond it chunks are sampled and the report says so; also caps the record walk of MCAPs without a summary), --nan-budget-mb (float bytes read per file for NaN counts), --block-kb (cached read block).

Reports go to ./reports/<repo>__<timestamp>.{json,md} (override with --report-dir, --report-name) and include, per file, how many bytes were read and in how many requests. Exit code is 1 when any file failed, so it is CI-friendly.

Modules

• phys_ai_dataprofiler/remote/range_file.py: HfRangeFile (seekable file over Hub byte ranges, block cache, parallel fetch_many) and LocalRangeFile (same API on disk).
• phys_ai_dataprofiler/remote/inspector.py: inspect_mcap_remote / inspect_mat_remote, returning the same McapInspectionResult, MatInspectionResult and RichProfile types as the full-read inspectors. Run it on a local file with python -m phys_ai_dataprofiler.remote.inspector file.mcap to see what it extracts.
• phys_ai_dataprofiler/inspectors/rich.py exposes timing_stats and profile_mcap_channels, shared by both paths so remote and local statistics are identical.

from phys_ai_dataprofiler.connectors.huggingface import HuggingFaceConnector
from phys_ai_dataprofiler.remote.inspector import inspect_mcap_remote, inspect_mat_remote, sniff_format

hf = HuggingFaceConnector("org/dataset", token="hf_xxx")
hf.connect()
selected, skipped = hf.select_files(hf.list_files(), max_files=10)
for f in selected:
    rf = hf.open_range(f, block_size=64 * 1024)
    fmt = sniff_format(rf, f.path)
    result, profile = inspect_mcap_remote(rf) if fmt == "mcap" else inspect_mat_remote(rf)
    print(f.path, rf.bytes_fetched, "of", rf.size, "bytes read")

Development

git clone https://github.com/akrishni/phys-ai-dataprofiler
cd phys-ai-dataprofiler
pip install -e ".[mat73,rosbag,dev]"
python tests/run_all.py                  # every test script, with a summary
python tests/run_all.py --hub            # also profile small public Hub datasets
python tests/test_rosbag_inspector.py    # or run one script on its own

CI (.github/workflows/tests.yml) runs tests/run_all.py on every push and pull request, on Python 3.10 to 3.13 on Linux and 3.13 on macOS, and builds the package and smoke-tests the installed CLI. The Hub checks need network access and are only run on demand: Actions tab, "tests", "Run workflow", tick "hub".

phys_ai_dataprofiler/
  connectors/huggingface.py   Hub connector and the default CLI command
  cli.py                      check, compare, requirements subcommands
  requirements.py evaluate.py compare.py   requirements files, verdicts, comparisons
  presets/                    built-in requirements files
  remote/                     byte-range readers: MCAP, MAT, Parquet, ROS bags
  inspectors/                 full-read inspectors for local files
tests/  samples/  scripts/  docs/

License and citation

MIT License, copyright (c) 2026 Anand Krishniyer. See LICENSE.

If you use this tool in research, please cite it; GitHub's "Cite this repository" button (from CITATION.cff) gives a ready-made reference.