DatasetLint

DatasetLint is a lightweight, local-first dataset validation, dataset QA, and linting toolkit for robotics and physical AI datasets. It checks dataset structure, metadata, timestamps, labels, calibration, trajectories, simple distribution stats, dataset diffs, and adapter compatibility before bad data reaches training or evaluation.

It runs locally on folder-based datasets with Python, CSV, and JSON. It does not require robots, simulators, GPUs, ROS, cloud services, or model runtimes.

Documentation: DatasetLint docs

Docs map:

Current Limitations

• Deep validation works best on the native DatasetLint folder format.
• Adapters provide manifest inspection, manifest export, and adapter validation for external formats.
• nuScenes, MCAP, ROS bag, Waymo, and Hugging Face have bounded --deep adapter modes: nuScenes inspects referenced camera/lidar/radar payload headers, MCAP and ROS bag parse channel/topic metadata, Waymo parses frame/label/calibration metadata, and Hugging Face samples rows. If a requested deep parser cannot parse the input, validation fails instead of reporting a valid deep pass.
• Hugging Face --deep uses bounded row sampling rather than scanning entire remote datasets by default.
• DatasetLint is not a dataset management platform, model evaluation framework, simulator, replay tool, or data host.
• Large-dataset performance has not been benchmarked yet.
• Adapter support is strongest at metadata/manifest validation; some payload formats still need format-specific decoders for full semantic checks.

Try In 60 Seconds

After installing from source, run one passing dataset, one failing dataset, and one machine-readable report:

datasetlint --version
datasetlint examples/minimal_dataset
datasetlint examples/bad_dataset
datasetlint examples/bad_dataset --format json
datasetlint examples/bad_dataset --format html > report.html
datasetlint diff examples/minimal_dataset examples/bad_dataset --fail-on-regression

examples/minimal_dataset should pass. examples/bad_dataset and the diff command should exit non-zero because they intentionally contain robotics data quality problems.

Common next steps:

datasetlint DATASET_PATH --checks calibration,labels
datasetlint DATASET_PATH --config DATASET_PATH/datasetlint.yaml --fail-on warning
datasetlint validate DATASET_PATH --adapter auto --format json
datasetlint validate logs/run.mcap --adapter mcap --deep --format json

Use the docs when you need the exact workflow:

• Getting Started for the first local run
• Configuration for per-rule enable/disable and severity overrides
• Adapters for MCAP, ROS bag, Argoverse 2, LeRobot, COCO, KITTI, nuScenes, Waymo, Hugging Face, and plugins
• Troubleshooting for reading failures and fixing noisy checks

Why This Exists

Robotics datasets often fail in quiet ways:

• dataset schema drift between collection, labeling, and training
• broken or duplicated timestamps
• missing metadata, calibration, sensor files, or referenced frames
• inconsistent labels and track IDs
• silent frame drops or distribution shifts
• adapter-specific ingestion problems discovered too late

DatasetLint catches those issues at the dataset folder boundary so teams can fail fast in local development and CI.

What DatasetLint Catches

Concrete native-folder checks include:

• missing metadata
• broken file references
• duplicate timestamps
• timestamp gaps
• sensor sync gaps
• missing calibration
• invalid camera intrinsics
• non-normalized quaternion
• invalid label geometry
• duplicate track ID at same timestamp
• unrealistic trajectory speed
• dataset diff regression

Who It Is For

• robotics ML engineers validating training and evaluation data
• physical AI teams maintaining local dataset collections
• dataset maintainers reviewing schema and metadata quality
• researchers sharing small reproducible datasets
• CI users blocking bad dataset changes before merge

Current Capabilities

DatasetLint v0.0.2 supports deep rule validation for the native folder dataset format and normalized manifest workflows for common external dataset formats. Current capabilities include:

• required metadata.json and calibration.json
• empty CSV files, broken referenced paths, and duplicate filenames
• metadata schema, declared sensors, duration consistency, and version presence
• monotonic, duplicate, and large-gap timestamps
• camera, IMU, GPS, and generic sensor columns
• camera dimensions, expected sensor rates, and likely missing frames
• sensor time overlap, start offsets, pairwise sync gaps, burst gaps, and jitter
• calibration intrinsics, extrinsics, and quaternion normalization
• label columns, confidence, geometry, timestamp range, class switches, duplicate tracks, short tracks, missing labels, box jumps, and size changes
• trajectory columns, finite values, speed, acceleration, yaw range, and stationary motion
• dataset statistics and folder-to-folder diffs
• normalized adapter manifests for generic folders, COCO, KITTI, Argoverse 2, LeRobot, nuScenes, Waymo, ROS bag, MCAP, and Hugging Face datasets
• adapter validation, inspection, discovery, and manifest export commands
• shared manifest-rule summaries for decoded adapter records, including frame references, timestamp consistency, sensor links, calibration shape, annotation links, and split references where the adapter exposes those records
• bounded deep adapter diagnostics for nuScenes payload headers, MCAP/ROS bag timestamp metadata, Waymo frame metadata, and sampled Hugging Face rows
• single-file adapter roots such as .bag, .mcap, and .tfrecord resolve relative frame/file records from the containing directory

Installation

DatasetLint requires Python 3.10 or newer. Install the latest release from PyPI:

python -m pip install datasetlint

For development, install from source:

git clone https://github.com/gagandeepreehal/datasetlint.git
cd datasetlint
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
python -m pip install -e ".[dev,docs]"

Use python3.10, python3.11, or python3.12 if that is the executable name on your machine. The macOS system python3 may be Python 3.9, which is too old for this project.

Adapter extras are opt-in so the base install stays lightweight:

python -m pip install -e ".[adapters]"
python -m pip install -e ".[hf]"
python -m pip install -e ".[mcap]"
python -m pip install -e ".[ros]"
python -m pip install -e ".[nuscenes]"
python -m pip install -e ".[waymo]"
python -m pip install -e ".[all-adapters]"

Quickstart

Validate the passing example dataset:

datasetlint lint examples/minimal_dataset
datasetlint examples/minimal_dataset

Expected result:

DatasetLint report for .../examples/minimal_dataset: passed with 0 issue(s) (error=0, warning=0, info=0).

Run a focused check group:

datasetlint examples/minimal_dataset --checks labels
datasetlint examples/minimal_dataset --checks sync

Render machine-readable output:

datasetlint report examples/minimal_dataset --out report.json
datasetlint examples/minimal_dataset --format json
datasetlint examples/minimal_dataset --format markdown
datasetlint examples/minimal_dataset --format html > report.html

Inspect the intentionally failing dataset:

datasetlint examples/bad_dataset

examples/bad_dataset exits non-zero because it contains missing files, invalid calibration, timestamp issues, label problems, and trajectory anomalies.

CLI Usage

DatasetLint exposes one console script:

datasetlint --help
datasetlint --version

Validate a dataset:

datasetlint lint DATASET_PATH
datasetlint DATASET_PATH
datasetlint DATASET_PATH --checks labels,sync
datasetlint DATASET_PATH --config DATASET_PATH/datasetlint.yaml
datasetlint DATASET_PATH --adapter folder
datasetlint DATASET_PATH --adapter auto
datasetlint DATASET_PATH --format console
datasetlint DATASET_PATH --format json
datasetlint DATASET_PATH --format markdown
datasetlint DATASET_PATH --format html
datasetlint DATASET_PATH --fail-on warning

Write validation reports:

datasetlint report DATASET_PATH --out report.json
datasetlint report DATASET_PATH --out report.md
datasetlint report DATASET_PATH --out report.html

Compute statistics:

datasetlint stats DATASET_PATH
datasetlint stats DATASET_PATH --format json
datasetlint stats DATASET_PATH --format markdown

Compare two datasets:

datasetlint diff OLD_DATASET NEW_DATASET
datasetlint diff OLD_DATASET NEW_DATASET --format json
datasetlint diff OLD_DATASET NEW_DATASET --fail-on-regression

Inspect adapter detection:

datasetlint adapters DATASET_PATH
datasetlint adapters list
datasetlint adapters detect DATASET_PATH
datasetlint adapters DATASET_PATH --format json

Inspect, validate, or export normalized manifests for common formats:

datasetlint inspect DATASET_PATH --adapter coco
datasetlint inspect DATASET_PATH --auto-detect
datasetlint validate DATASET_PATH --adapter kitti
datasetlint validate DATASET_PATH --adapter nuscenes --deep
datasetlint validate DATASET_PATH --adapter mcap --deep
datasetlint inspect DATASET_PATH --adapter waymo --deep --max-rows 1000
datasetlint validate hf://namespace/dataset --adapter huggingface --split train --deep --max-rows 1000
datasetlint export-manifest DATASET_PATH --adapter nuscenes --output manifest.json

Exit codes:

• 0: command completed and did not meet the configured failure threshold
• 1: validation failed the --fail-on threshold, adapter validation failed, or diff regressions were found with --fail-on-regression
• 2: invalid usage, unknown check group, bad adapter, or invalid config

Python API Usage

from datasetlint import compare_datasets, compute_dataset_stats, lint_dataset
from datasetlint.adapters import detect_adapters, get_adapter, load_dataset, validate_dataset

report = lint_dataset("examples/minimal_dataset")
print(report.summary())
print(report.count_by_severity())
print(report.to_markdown())

label_report = lint_dataset("examples/minimal_dataset", checks="labels")
sync_report = lint_dataset("examples/minimal_dataset", checks=["sync"])

stats = compute_dataset_stats("examples/minimal_dataset")
print(stats.frame_counts)

diff = compare_datasets("examples/minimal_dataset", "examples/bad_dataset")
print(diff.summary)

detections = detect_adapters("examples/minimal_dataset")
adapter = get_adapter("examples/minimal_dataset", "auto")

manifest = load_dataset("tests/fixtures/coco_dataset", adapter="coco")
validation = validate_dataset("tests/fixtures/kitti_object", adapter="kitti")

Public imports from datasetlint are lint_dataset, compare_datasets, compute_dataset_stats, Issue, LintConfig, LintReport, DatasetStats, and DatasetDiffReport.

Examples

See examples/README.md for commands and expected outcomes.

Generated sample reports are committed under examples/reports/:

• examples/reports/minimal_report.json
• examples/reports/bad_report.json
• examples/reports/bad_report.md

Configuration

DatasetLint uses defaults when no config is provided. A dataset can include datasetlint.yaml, or the CLI can receive --config path/to/datasetlint.yaml.

Supported keys:

timestamp_gap_threshold_sec: 0.5
frequency_tolerance_fraction: 0.30
missing_frame_gap_multiplier: 1.5
max_speed_mps: 70
max_accel_mps2: 12
stationary_distance_threshold_m: 0.05
duration_tolerance_sec: 1.0
label_max_position_jump_px: 200
label_max_size_change_ratio: 3.0
label_min_track_length: 3
label_class_switch_threshold: 0
max_pairwise_sync_gap_sec: 0.05
min_overlap_ratio: 0.8
frequency_jitter_ratio: 0.2
frame_count_drop_ratio_warning: 0.1
duration_drop_ratio_warning: 0.1
issue_regression_severity: warning
rules:
  enabled: null
  disabled: []
  severity: {}
expected_sensor_rates:
  camera_front: 10
  camera_rear: 10
  imu: 100
  gps: 10

The config reader supports normal YAML lists and nested maps. Unknown keys fail validation so stale configs do not silently pass. Rule policy can live in config:

rules:
  enabled:
    - calibration
    - labels
  disabled:
    - check_sensor_frequency
  severity:
    check_large_timestamp_gaps: info

Reports

Validation reports include:

• summary pass/fail state
• issue severity: error, warning, or info
• check name
• file path and 1-based CSV row when available
• message and structured metadata
• stats such as issue counts, sensor counts, and sync diagnostics

Output formats are console, JSON, Markdown, and static HTML:

datasetlint examples/minimal_dataset --format console
datasetlint examples/minimal_dataset --format json
datasetlint examples/minimal_dataset --format markdown
datasetlint examples/minimal_dataset --format html > report.html
datasetlint report examples/bad_dataset --out report.html

Issue row numbers use spreadsheet-style rows: the CSV header is row 1 and the first data row is row 2.

CI Usage

Minimal GitHub Actions example:

name: Dataset Lint
on:
  pull_request:
  push:
    branches: [main]

jobs:
  datasetlint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: python -m pip install -e ".[dev]"
      - run: datasetlint examples/minimal_dataset

Use --fail-on warning for stricter validation, or datasetlint diff OLD_DATASET NEW_DATASET --fail-on-regression when comparing dataset revisions.

More CI templates, including report artifacts and dataset diffs, are in docs/ci.md.

Supported Formats And Adapters

Roadmap

Near term:

• stronger dataset diff coverage
• more targeted example datasets
• CI templates for common repository layouts
• clearer rule plugin examples

Medium term:

• modality-specific decoding beyond current metadata manifests: MCAP and ROS bag message payloads, Waymo image/lidar payload bytes, richer Argoverse 2/LeRobot payload semantics, full nuScenes devkit/map/point-cloud semantics, and dataset-specific Hugging Face row schemas
• conversion helpers from normalized manifests to the native lintable folder format
• richer sensor synchronization checks
• richer static HTML report styling while keeping reports dependency-free

Long term:

• rule plugin system
• benchmark sample datasets
• performance profiling on larger logs
• adapter compatibility test suite

Non-goals:

• replacing dataset version control systems
• storing or hosting datasets
• running model evaluation
• simulating robotics environments

Contributing

Read CONTRIBUTING.md for local setup, tests, linting, type checks, docs, and guidance for adding rules or adapters.

License

DatasetLint is released under the MIT License.