seatrialkit

Scoring functions for marine autonomy trials, computed from ground-truth vehicle trajectories.

Motivation

seatrialkit was built to score marine autonomy trials by consistent published metrics, regardless of the setting, in any simulator or at sea.

Each autonomy trial produces a time series of the vehicle's position and heading. Station-keeping error, distance to each waypoint and time to arrival are all properties of that time series and the trial's goal. seatrialkit therefore defines each score as a function of exactly two inputs: the recorded track and a scenario file that describes the trial.

Defining scores on these two inputs has three results:

• One implementation for every source of trial data. Any simulator, and any field trial with an independent position reference, can produce the track, and seatrialkit scores each track with identical code.
• Scores that anyone can recompute. A reported score can be reproduced from the track and the scenario file without access to the simulator, middleware or autonomy software used in the trial.
• Open definitions. Each metric has a written formula, each free parameter is an argument stored with the result, and each metric taken from published work is tested against its original implementation. The first such metrics are the station-keeping and wayfinding scores of the Virtual RobotX (VRX) simulation [1, 2].

Interpreting a score requires knowing the environmental conditions of the trial. A station-keeping error of 2 m reflects one level of control performance in slack water and another in a 1.5 m/s tidal current. A planner given an accurate current forecast also faces a different task from a planner given an inaccurate one. For this reason the scenario file is part of every score: it records each environmental input to the trial (currents, wind, waves, bathymetry), the source of each input (an observation, a hindcast or a forecast), and its role, either the conditions acting on the vehicle or the information available to the vehicle's planner.

Long Horizon Observatory built seatrialkit for its own evaluation work. The Observatory prepares environmental data, including ocean forecasts, hindcasts and bathymetry, for high-resolution simulation. It integrates established open-source simulators and autonomy software with that data, and contributes to those projects, so that existing autonomy software can be run under the recorded environmental conditions of specific days and places. Comparing those runs with one another, and with field trials conducted with partner operators, requires scores computed identically in every setting. The Observatory plans to use seatrialkit to score these trials, to measure how error in forecast environmental inputs changes the scores, and to publish the definitions and code behind every score it reports.

Out of scope: simulation, vehicle models, environmental data processing and running autonomy stacks.

Metrics

Formulas, units and reporting rules are in the documentation at seatrialkit.docs.lhzn.io (source in docs/source/). The combined pose error is E = d + 0.75^d * h, where d is the distance to the goal in meters and h the heading error in radians, as in the VRX scoring plugins [2]. The VRX paper [1] names the tasks; the formulas are stated in the code. The test suite compares seatrialkit with line-by-line transliterations of the VRX update loops on random trajectories; the two agree to a relative tolerance of 1e-12. Trajectories replayed through the VRX scoring plugins themselves, built from the same commit and run in Gazebo Harmonic, give the same scores to within 1e-14 (see Validation in the documentation).

Scenarios

A scenario file (YAML) names the region, vehicle, mission, timing windows, scoring set and random seeds. It is validated against a JSON Schema shipped in the package (seatrialkit/scenario.schema.json, version 0). Scenario identifiers follow the convention of the CommonOcean benchmarks [3]. scenario_hash returns the SHA-256 of the scenario in canonical JSON, which is independent of key order. Two idealized examples are in examples/scenarios/.

Installation

pip install seatrialkit              # metrics only
pip install "seatrialkit[scenario]"  # adds scenario loading and validation

Requires Python 3.11 or later.

Example

import numpy as np
import seatrialkit as tk

t = np.arange(0, 300, 1 / 60)
traj = tk.Trajectory(t, x=0.3 * np.sin(t / 20), y=0.2 * np.cos(t / 15), yaw=np.full_like(t, np.pi / 2))
result = tk.score_stationkeeping(traj, goal_x=0.0, goal_y=0.0, goal_yaw=np.pi / 2, watch_radius_m=2.0)
print(result.as_dict())

Dependencies

• NumPy [6]: all metric computations.
• PyYAML and jsonschema (optional extra scenario): scenario loading and schema validation.

Related projects (not dependencies):

• osrf/vrx (Apache-2.0) [2]: source of the station-keeping and wayfinding definitions.
• CommonOcean: benchmark identifiers and scenario conventions for motion planning on water.

Status

Version 0.1.0, pre-release. Metric definitions and the scenario schema may change before 1.0. Record the seatrialkit version with every reported score. A score computed in simulation describes behavior in that simulation, not field performance.

References

1. B. Bingham, C. Agüero, M. McCarrin, J. Klamo, J. Malia, K. Allen, T. Lum, M. Rawson and R. Waqar, "Toward Maritime Robotic Simulation in Gazebo," in OCEANS 2019 MTS/IEEE Seattle, 2019, pp. 1-10. doi:10.23919/OCEANS40490.2019.8962724
2. Open Source Robotics Foundation, VRX: Virtual RobotX, source code, branch jazzy, commit 7609d1b, scoring plugins vrx_gz/src/StationkeepingScoringPlugin.cc and vrx_gz/src/WayfindingScoringPlugin.cc. Apache-2.0. github.com/osrf/vrx, read 2026-11-12.
3. H. Krasowski and M. Althoff, "CommonOcean: Composable Benchmarks for Motion Planning on Oceans," in 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), 2022, pp. 1676-1682. doi:10.1109/ITSC55140.2022.9921925
4. D. Perille, A. Truong, X. Xiao and P. Stone, "Benchmarking Metric Ground Navigation," in 2020 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), 2020, pp. 116-121. doi:10.1109/SSRR50563.2020.9292572
5. X. Xiao et al., "Autonomous Ground Navigation in Highly Constrained Spaces: Lessons Learned From the Benchmark Autonomous Robot Navigation Challenge at ICRA 2022," IEEE Robotics & Automation Magazine, vol. 29, no. 4, pp. 148-156, 2022. doi:10.1109/MRA.2022.3213466. Current rules: BARN Challenge 2026.
6. C. R. Harris et al., "Array programming with NumPy," Nature, vol. 585, pp. 357-362, 2020. doi:10.1038/s41586-020-2649-2

License

MIT. Copyright Long Horizon Observatory.