English | 简体中文

slam-regression-ci

Variance-aware CI regression gates for SLAM and odometry — fingerprinted baselines, coverage checks, repeated-run statistics, and multi-sequence suites. Compare a candidate trajectory against a recorded baseline, apply configurable thresholds, and fail CI when localization accuracy got worse — with a human-readable Markdown report and a machine-readable JSON report.

evo measures trajectory accuracy; slam-regression-ci decides whether a code change caused a meaningful regression. It ships a minimal native TUM evaluator (parity-checked against evo; evo remains the richer evaluation and plotting tool).

SLAM and odometry code changes can quietly degrade accuracy. Evaluation tools like evo compute ATE/RPE, but they don't decide "is this PR a regression?". Hand-writing that policy per repository is tedious and error-prone. This tool does exactly that, and nothing else:

baseline JSON (committed) + ground truth + candidate estimate
        │
        ▼
ATE RMSE / RPE thresholds ──► exit 0 (pass) or exit 1 (regression)
        │
        ▼
report.md (for humans) + report.json (for CI/automation)

Early-stage project seeking real-world feedback — try it on your repository and open an issue.

Installation

Requires Python 3.8+ (works on ROS1 Noetic / Ubuntu 20.04) and only numpy + PyYAML at runtime.

pip install git+https://github.com/pyy52/slam-regression-ci.git@v0.2.1

Or try it without installing:

git clone https://github.com/pyy52/slam-regression-ci.git
cd slam-regression-ci && pip install .

60-second quickstart

The repository ships deterministic synthetic examples (a circle trajectory, a noisy "good" estimate, and a degraded candidate):

# 1) Passes: the baseline compared against itself (~0% change), exit code 0
slam-regression compare \
  --baseline examples/baseline_metrics.json \
  --reference examples/reference.tum \
  --estimate examples/baseline.tum \
  --config examples/regression.yaml

# 2) Fails: degraded candidate (ATE +282%), exit code 1
slam-regression compare \
  --baseline examples/baseline_metrics.json \
  --reference examples/reference.tum \
  --estimate examples/candidate_degraded.tum \
  --config examples/regression.yaml \
  --report report.md --json report.json
echo $?

For your own project, record a baseline once and commit it:

# After a known-good run of your SLAM system:
slam-regression record \
  --reference ground_truth.tum \
  --estimate my_slam_output.tum \
  --json baselines/room1_baseline.json

SLAM pipelines are often not fully deterministic (RANSAC, threading, GPUs). Record several runs and the baseline becomes a distribution instead of a single number:

slam-regression record \
  --reference ground_truth.tum \
  --estimate run1.tum run2.tum run3.tum \
  --json baselines/room1_baseline.json    # schema v2: median / MAD / std

# Compare the median of repeated candidate runs; optionally fail when the
# candidate leaves the baseline's natural run-to-run spread:
slam-regression compare \
  --baseline baselines/room1_baseline.json \
  --reference ground_truth.tum \
  --estimate cand1.tum cand2.tum \
  --config regression.yaml

Then gate every change against it:

slam-regression compare \
  --baseline baselines/room1_baseline.json \
  --reference ground_truth.tum \
  --estimate my_slam_output.tum \
  --config regression.yaml \
  --report regression_report.md --json regression_report.json

Example output

ATE RMSE:
  baseline:  0.017247 m
  candidate: 0.065853 m
  change: +281.81%
  threshold: +10.00%
RPE translation RMSE:
  baseline:  0.025050 m
  candidate: 0.074284 m
  change: +196.60%
  threshold: +10.00%
STATUS: FAIL

Exit codes: 0 pass, 1 regression, 2 input/config error.

Configuration

metrics:
  # A metric fails when ANY configured rule is exceeded, so configure only
  # the rules you want. Improvements always pass relative/absolute rules.
  ate_rmse:
    max_relative_regression_percent: 10   # relative increase over baseline
    max_absolute_regression: 0.03         # ...and at most 3 cm in absolute terms
    # max_value: 0.25                     # absolute ceiling on the value itself
    # max_mad_multiples: 3                # multi-run baselines: fail if beyond
    #                                     # median + 3*MAD of the recorded runs
  rpe_translation_rmse:
    max_relative_regression_percent: 10

association:
  max_timestamp_diff: 0.01   # seconds; each ref pose matches its NEAREST est pose

alignment:
  enabled: true              # Umeyama alignment of estimate to reference
  correct_scale: false       # also estimate a uniform scale (monocular)

coverage:
  # Optional gates on how much of the reference the candidate tracks
  # (catches early tracking loss that would otherwise look like a good ATE).
  # For keyframe-based estimates (ORB-SLAM etc.) prefer min_time_coverage_ratio:
  # a keyframe output matches only a fraction of reference poses by design
  # (low matched_pose_ratio is normal), but it still spans the whole timeline.
  min_matched_pose_ratio: 0.95
  min_time_coverage_ratio: 0.95

rpe_delta: 1                 # frame gap for the relative pose error

Every key is optional: omitting metrics keeps the default gates, omitting the whole file uses defaults for everything. Unknown keys and values are rejected with actionable error messages. Baselines record sha256 fingerprints of their inputs; comparing against a changed ground-truth file or with different metric-semantic settings is a hard error unless --allow-incompatible-baseline is given.

Use it as a CI gate in your repository

# .github/workflows/slam-regression.yml in YOUR repository
name: SLAM regression
on: [pull_request]

jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install git+https://github.com/pyy52/slam-regression-ci.git@v0.2.1
      - name: Check trajectory regression
        run: |
          slam-regression compare \
            --baseline baselines/room1_baseline.json \
            --reference data/room1_gt.tum \
            --estimate results/room1_est.tum \
            --config regression.yaml \
            --report regression_report.md --json regression_report.json
      - name: Attach report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: regression-report
          path: |
            regression_report.md
            regression_report.json

Commit the baseline JSON (baselines/room1_baseline.json) once, and every pull request that makes localization worse will fail — with an attached Markdown report explaining exactly which metric moved.

Real SLAM systems are evaluated on more than one sequence. Define a suite and get one combined verdict, with failed sequences ordered worst-regression first:

# suite.yaml
sequences:
  - name: room1
    baseline: baselines/room1.json
    reference: data/room1_gt.tum
    estimate: results/room1_est.tum
  - name: corridor
    baseline: baselines/corridor.json
    reference: data/corridor_gt.tum
    estimate: [results/corridor_run1.tum, results/corridor_run2.tum]  # median of runs

suite_policy:
  max_failed_sequences: 0   # overall fails if more than this many sequences fail

slam-regression suite --config suite.yaml --report suite_report.md --json suite_report.json

Prefer a one-line action instead? The repository is also a composite action:

      - uses: pyy52/slam-regression-ci@v0
        with:
          baseline: baselines/room1_baseline.json
          reference: data/room1_gt.tum
          estimate: results/room1_est.tum

See what the report looks like before installing: sample report.

Docker

docker build -f docker/Dockerfile -t slam-regression-ci .
docker run --rm -v "$PWD/examples:/data" slam-regression-ci compare \
  --baseline /data/baseline_metrics.json \
  --reference /data/reference.tum \
  --estimate /data/candidate_degraded.tum \
  --config /data/regression.yaml

Metric trustworthiness

ATE/RPE are implemented on top of numpy following the TUM RGB-D benchmark conventions, and CI verifies parity against evo (the reference implementation) on the bundled examples within 1e-6 relative tolerance (parity workflow). evo remains the recommended tool for advanced evaluation; this project deliberately stays minimal.

Supported input formats

• TUM text files: timestamp tx ty tz qx qy qz qw (one pose per line, # comments allowed, quaternion order x y z w) — the format produced by the TUM RGB-D benchmark tools and consumed by evo.

Timestamp association is one-to-one nearest: each reference pose is paired with its nearest estimate pose not already in use, within max_timestamp_diff. On real data this yields the same pair count as evo's own synchronization; small metric differences (≤0.3% on a real ORB-SLAM keyframe run) can arise from the selection order (evo picks each estimate pose's nearest reference, we pick each reference pose's nearest estimate). The original TUM benchmark script used a similar one-to-one walk but took the first pose within tolerance instead of the nearest.

Limitations

• Trajectory input: TUM format only (KITTI poses planned).
• Metrics: ATE RMSE and RPE translation RMSE with consecutive frames (delta = 1); no per-axis breakdowns, no plots.
• Baselines are tied to a specific ground-truth file and dataset; changing the dataset means recording a new baseline.
• Reports are written to files; there is no PR-comment bot yet.

Roadmap

• PyPI package — on PyPI
• KITTI pose file support
• Additional metrics (RPE with configurable delta units, rotation error)
• Optional PR comment with the regression summary
• Baseline refresh helper (record --update policy)

Contributing

Issues and pull requests are welcome — see CONTRIBUTING.md. Note that this project is maintained with AI assistance for routine engineering work; all changes are reviewed before merge.

License

MIT