betterosi - a python library for reading and writing open-simulation-interface files

A python library for reading and writing ASAM OSI (Open-Simulation-Interace) files (either .osi binary traces or MCAP files) using protobuf-py for typed, ergonomic protobuf message classes.

• Supports writing and reading either mcap or osi files with betterosi.Writer and betterosi.read.
• View OSI or MCAP file containing OSI GroundTruth betterosi-viewer <filepath.mcap / filepath.osi>(adapted from esmini)
• Convert osi to mcap with betterosi-to-mcap <filepath to osi>.

The library uses code from esmini (betterosi/viewer.py) under MPL 2.0 license. Reading and writing of .osi traces and MCAP files is implemented in betterosi/io.py.

The library uses buf with protoc-gen-py to generate typed Python code from the protobuf definitions of open-simulation-interface.

Since OSI and esmini are under MPL, also this repository is published under MPL-2.0 license.

OSI Version Support

betterosi supports multiple ASAM OSI versions simultaneously (currently 3.7.0 and 3.8.0).

By default, OSI 3.8.0 is used:

import betterosi

gt = betterosi.GroundTruth(...)

To target a specific OSI version, import the corresponding version namespace:

from betterosi import v3_7_0 as betterosi  # uses OSI 3.7.0

or

from betterosi import v3_8_0 as betterosi  # uses OSI 3.8.0

Differences to Standard OSI

The proto definitions extend standard OSI definitions in the following ways:

• Add MapAsamOpenDrive Message: Packages the XML content of an ASAM OpenDRIVE map in a proto Message

See omega-prime for details.

Install

pip install betterosi

Create an OSI or MCAP trace

To create an OSI or MCAP trace, you need to use betterosi.Writer. After creating the OSI Message of your desire, just add it to the Writer as shown in the examples below for either MCAP traaces or OSI traces.

import betterosi

with betterosi.Writer("test.mcap") as writer:
    gt = betterosi.GroundTruth(...)
    writer.add(gt)

with betterosi.Writer("test.osi") as writer:
    sv = betterosi.SensorView(...)
    writer.add(sv)

Below a full example is given which creates three files, and MCAP trace and and OSI trace with GroundTruth messages and a MCAP trace of SensorViews. If you use the code, you obviously just need one of the writers.

import betterosi

NANOS_PER_SEC = 1_000_000_000


with (
    betterosi.Writer("test.mcap") as writer_mcap,
    betterosi.Writer("test.osi") as writer_osi,
    betterosi.Writer("test_sv.mcap") as writer_sv,
):
    moving_object = betterosi.MovingObject(
        id=betterosi.Identifier(value=42),
        type=betterosi.MovingObjectType.UNKNOWN,
        base=betterosi.BaseMoving(
            dimension=betterosi.Dimension3D(length=5, width=2, height=1),
            position=betterosi.Vector3D(x=0, y=0, z=0),
            orientation=betterosi.Orientation3D(roll=0.0, pitch=0.0, yaw=0.0),
            velocity=betterosi.Vector3D(x=1, y=0, z=0),
        ),
    )
    gt = betterosi.GroundTruth(
        version=betterosi.InterfaceVersion(
            version_major=3, version_minor=8, version_patch=0
        ),
        timestamp=betterosi.Timestamp(seconds=0, nanos=0),
        moving_object=[moving_object],
        host_vehicle_id=betterosi.Identifier(value=0),
    )
    sv = betterosi.SensorView(
        version=betterosi.InterfaceVersion(
            version_major=3, version_minor=8, version_patch=0
        ),
        timestamp=betterosi.Timestamp(seconds=0, nanos=0),
        global_ground_truth=gt,
        host_vehicle_id=betterosi.Identifier(value=0),
    )
    # Generate 1000 OSI messages for a duration of 10 seconds
    for i in range(1000):
        total_nanos = i * 0.01 * NANOS_PER_SEC
        gt.timestamp.seconds = int(total_nanos // NANOS_PER_SEC)
        gt.timestamp.nanos = int(total_nanos % NANOS_PER_SEC)
        moving_object.base.position.x += 0.5
        sv.timestamp = gt.timestamp

        writer_mcap.add(gt)
        writer_osi.add(gt)
        writer_sv.add(sv)

When writing MCAP messages you can specifiy the topic in the writer and the add function by setting the topic argument. When reading such files, pass the topic(s) as a list to the mcap_topics argument, e.g. mcap_topics=["ground_truth"]. By default, all channels with an OSI schema are read.

Read OSI and MCAP

With betterosi.read you can read an mcap or osi trace. read returns a generator. With the following code, you can get a list of the GroundTruth messages from a trace, even if the GroundTruth are nested inside SensorViews. It works the same for OSI traces (see below for how the message type of an .osi file is determined).

import betterosi

ground_truths = list(betterosi.read("test_sv.mcap", return_ground_truth=True))
print([len(ground_truths), ground_truths[0]])

Above code prints the following:

[1000, GroundTruth(version=InterfaceVersion(version_major=3, version_minor=8, version_patch=0), timestamp=Timestamp(seconds=0, nanos=0), host_vehicle_id=Identifier(value=0), moving_object=[MovingObject(id=Identifier(value=42), base=BaseMoving(dimension=Dimension3d(length=5.0, width=2.0, height=1.0), position=Vector3d(x=0.5, y=0.0, z=0.0), orientation=Orientation3d(roll=0.0, pitch=0.0, yaw=0.0), velocity=Vector3d(x=1.0, y=0.0, z=0.0)), type=MovingObject.Type.UNKNOWN)])]

If you want a list of the sensor views directly:

import betterosi

sensor_views = betterosi.read("test_sv.mcap", return_sensor_view=True)
print(next(sensor_views))

The above prints:

SensorView(version=InterfaceVersion(version_major=3, version_minor=8, version_patch=0), timestamp=Timestamp(seconds=0, nanos=0), global_ground_truth=GroundTruth(version=InterfaceVersion(version_major=3, version_minor=8, version_patch=0), timestamp=Timestamp(seconds=0, nanos=0), host_vehicle_id=Identifier(value=0), moving_object=[MovingObject(id=Identifier(value=42), base=BaseMoving(dimension=Dimension3d(length=5.0, width=2.0, height=1.0), position=Vector3d(x=0.5, y=0.0, z=0.0), orientation=Orientation3d(roll=0.0, pitch=0.0, yaw=0.0), velocity=Vector3d(x=1.0, y=0.0, z=0.0)), type=MovingObject.Type.UNKNOWN)]), host_vehicle_id=Identifier(value=0))

MCAP files store the message type of every channel, so reading them only needs the file name. .osi traces do not store it, and GroundTruth and SensorView messages cannot always be told apart from their bytes. read determines the type of an .osi file in this order:

1. the osi_message_type argument, if given,
2. the OSI trace file naming convention (<timestamp>_<type>_...osi, e.g. 20240101T120000Z_gt_380_700_1000_drive.osi) or a single gt / sv / ... part in the file name,
3. only with return_ground_truth / return_sensor_view: decoding the first message. If that is ambiguous, a ValueError asks you to pass osi_message_type.

import betterosi

any_osi_message = betterosi.read("test.osi", osi_message_type="GroundTruth")
any_osi_message = betterosi.read("test.mcap")

Generate library code

pip install protoc-gen-py buf-bin

python gen_protos.py

gen_protos.py compiles every version directory in osi-proto/<version>/ into betterosi/generated/<version>/ (with buf generate and an inline protoc-gen-py template, so no buf.yaml / buf.gen.yaml is needed) and builds the osi3_descriptor_set.pb used for MCAP schemas. Pass versions to generate only those, e.g. python gen_protos.py 3.8. The generated code is not committed; it is also created automatically by the build hook during uv sync / pip install.

Name

The library is called betterosi because it was formerly based on the python-betterproto2 library for handling protobufs.

LICENSE and Copyright

This code is published under MPL-2.0 license. It utilizes and modifies parts of esmini (betterosi/viewer.py) under MPL-2.0 and open-simulation-interface osi-proto/* under MPL-2.0.

Acknowledgements

This package is developed as part of the SYNERGIES project.

Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or European Climate, Infrastructure and Environment Executive Agency (CINEA). Neither the European Union nor the granting authority can be held responsible for them.

Notice