Cyber record offline parse tool
Project description
cyber_record
cyber_record is a cyber record file offline parse tool. You can use cyber_record
to read messages from record file, or write messages to the record file.
Quick start
First install "cyber_record" by the following command.
pip3 install cyber_record
// or update version
pip3 install cyber_record -U
python version
If protobuf prompt requires python>=3.7, you can install python3.7+ and switch default python version
sudo apt install python3.8
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 2
demo record
You can download a apollo demo record from demo_sensor_data_for_vision
Command line mode
You can easily get the information in the record file by the following command.
Info
cyber_record info
will output the statistics of the record file.
$ cyber_record info -f example.record.00000
record_file: example.record.00000
version: 1.0
begin_time: 2021-07-23 17:12:15.114944
end_time: 2021-07-23 17:12:15.253911
duration: 0.14 s
size: 477.55 KByte
message_number: 34
channel_number: 8
/apollo/planning , apollo.planning.ADCTrajectory , 1
/apollo/routing_request , apollo.routing.RoutingRequest , 0
/apollo/monitor , apollo.common.monitor.MonitorMessage , 0
/apollo/routing_response , apollo.routing.RoutingResponse , 0
/apollo/routing_response_history , apollo.routing.RoutingResponse , 1
/apollo/localization/pose , apollo.localization.LocalizationEstimate, 15
/apollo/canbus/chassis , apollo.canbus.Chassis , 15
/apollo/prediction , apollo.prediction.PredictionObstacles , 2
Echo
cyber_record echo
will print the message of the specified topic to the terminal.
$ cyber_record echo -f example.record.00000 -t /apollo/canbus/chassis
engine_started: true
speed_mps: 0.0
throttle_percentage: 0.0
brake_percentage: 0.0
driving_mode: COMPLETE_AUTO_DRIVE
gear_location: GEAR_DRIVE
header {
timestamp_sec: 1627031535.112813
module_name: "SimControl"
sequence_num: 76636
}
Or you can reference the cyber_record
in the python file by
from cyber_record.record import Record
recover
If you find record file is missing index, you can recover the file by cyber_record recover
.
It is best to backup the file before recover!!!
- Generate the file descriptor set. Must be executed in the
apollo
directory.
descriptor_set_out
is the descriptor file namemodules/drivers/proto/sensor_image.proto
the message define proto file
protoc --include_imports --descriptor_set_out=tmp modules/drivers/proto/sensor_image.proto
or you can use absolute path.
descriptor_set_out
is the descriptor file nameproto_path
the apollo home path/home/zero/01opencode/apollo/modules/drivers/proto/sensor_image.proto
proto file absolute path
protoc --include_imports --descriptor_set_out=tmp --proto_path=/home/zero/01opencode/apollo /home/zero/01opencode/apollo/modules/drivers/proto/sensor_image.proto
- Recover the record file.
broken.record
is the file need repair/apollo/sensor/camera/front_6mm/image
the topic of the need repair messagetmp
the descriptor file generated in the previous stepapollo.drivers.Image
the message type of the need repair message
cyber_record recover -f broken.record -t /apollo/sensor/camera/front_6mm/image -d tmp -m apollo.drivers.Image
Examples
Below are some examples to help you read and write messages from record files.
1. Read messages
You can read messages directly from the record file in the following ways.
from cyber_record.record import Record
file_name = "20210521122747.record.00000"
record = Record(file_name)
for topic, message, t in record.read_messages():
print("{}, {}, {}".format(topic, type(message), t))
The following is the output log of the program
/apollo/localization/pose, <class 'LocalizationEstimate'>, 1627031535246897752
/apollo/canbus/chassis, <class 'Chassis'>, 1627031535246913234
/apollo/canbus/chassis, <class 'Chassis'>, 1627031535253680838
Filter Read
You can also read messages filtered by topics and time. This will improve the speed of parsing messages.
def read_filter_by_both():
record = Record(file_name)
for topic, message, t in record.read_messages('/apollo/canbus/chassis', \
start_time=1627031535164278940, end_time=1627031535215164773):
print("{}, {}, {}".format(topic, type(message), t))
2. Parse messages
To avoid introducing too many dependencies, you can save messages by record_msg
.
pip3 install record_msg -U
record_msg
provides 3 types of interfaces
csv format
you can use to_csv
to format objects so that they can be easily saved in csv format.
f = open("message.csv", 'w')
writer = csv.writer(f)
def parse_pose(pose):
'''
save pose to csv file
'''
line = to_csv([pose.header.timestamp_sec, pose.pose])
writer.writerow(line)
f.close()
image
you can use ImageParser
to parse and save images.
image_parser = ImageParser(output_path='../test')
for topic, message, t in record.read_messages():
if topic == "/apollo/sensor/camera/front_6mm/image":
image_parser.parse(message)
# or use timestamp as image file name
# image_parser.parse(image, t)
lidar
you can use PointCloudParser
to parse and save pointclouds.
pointcloud_parser = PointCloudParser('../test')
for topic, message, t in record.read_messages():
if topic == "/apollo/sensor/lidar32/compensator/PointCloud2":
pointcloud_parser.parse(message)
# other modes, default is 'ascii'
# pointcloud_parser.parse(message, mode='binary')
# pointcloud_parser.parse(message, mode='binary_compressed')
3. Write messages
You can now also build record by messages. You can write pb_message by record.write
.
def write_message():
pb_map = map_pb2.Map()
pb_map.header.version = 'hello'.encode()
with Record(write_file_name, mode='w') as record:
record.write('/apollo/map', pb_map, int(time.time() * 1e9))
Its application scenario is to convert dataset into record files. Please note that it must be written in chronological order.
If you want to write raw message, you should first use Builder
to help convert raw data to pb_message.
image
You can write image to record file like below. ImageBuilder
will help you convert image to pb_image. encoding
should be rgb8
,bgr8
or gray
, y
.
def write_image():
image_builder = ImageBuilder()
write_file_name = "example_w.record.00002"
with Record(write_file_name, mode='w') as record:
img_path = 'test.jpg'
pb_image = image_builder.build(img_path, encoding='rgb8')
record.write('/apollo/sensor/camera/front_6mm/image',
pb_image,
int(time.time() * 1e9))
lidar
You can write image to record file like below. PointCloudBuilder
will help you convert pcd file to pb_point_cloud.
def write_point_cloud():
point_cloud_builder = PointCloudBuilder()
write_file_name = "example_w.record.00003"
with Record(write_file_name, mode='w') as record:
pcd_path = 'test.pcd'
pb_point_cloud = point_cloud_builder.build(pcd_path)
record.write('/apollo/sensor/lidar32/compensator/PointCloud2',
pb_point_cloud,
int(time.time() * 1e9))
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file cyber_record-0.1.12.tar.gz
.
File metadata
- Download URL: cyber_record-0.1.12.tar.gz
- Upload date:
- Size: 20.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.0 CPython/3.7.6
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | dc8d8c53f1600eef037377bab63a2ce80b714287a682053485e875031156a82d |
|
MD5 | 74248500f18bb9b1a62b13b4006a533f |
|
BLAKE2b-256 | 81d34c68ee19afc5aca50daf29e2e9e44295ac32f68316be08f36e96072bb41a |
File details
Details for the file cyber_record-0.1.12-py3-none-any.whl
.
File metadata
- Download URL: cyber_record-0.1.12-py3-none-any.whl
- Upload date:
- Size: 29.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.0 CPython/3.7.6
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | e20644488ad049a907a5c495966db58ba9218c4cab60402ab4e759337ff99138 |
|
MD5 | 312caee034df9e86d7b6fc2ba647f22e |
|
BLAKE2b-256 | f7a3cd0c6a610186b7bc1f1df841267416ebeeaca4541b0a93f9650f58934fbe |