ROS package used for serialising common ROS messages to a database or filesystem.
Project description
Topic Store
Topic Store is a ROS package for storing ROS messages to a database or filesystem.
Unlike ROS bags Topic Store adds flexibility by serialising all messages into a data hierarchy that's easily searchable with database queries and allows for remote storage.
You can also use Topic Store as a standalone python package to read and write data without a ROS installation!
Installation
ROS
# From source
cd catkin_ws/src
git clone https://github.com/RaymondKirk/topic_store
catkin build topic_store
# From apt
sudo apt install ros-melodic-topic-store # you need to add the L-CAS ros source
Installing without ROS
You can install topic_store as an independent python2/3 package without a working ROS installation.
# From source
git clone https://github.com/RaymondKirk/topic_store
cd topic_store/src
pip install --extra-index-url https://rospypi.github.io/simple/ -e .
# From PyPi
pip install --extra-index-url https://rospypi.github.io/simple/ topic-store
To install other dependencies i.e. ros_numpy
you can run the following
pip install --extra-index-url https://rospypi.github.io/simple/ ros_numpy
Usage
Create a scenario file by following the documentation in scenarios or by following the example file default_config.yaml. The example below will save a history of all messages sent to the ros log topic.
# Save to /path/to/your/scenario/file.yaml. This collection behaviour will save your log history.
context: "save_ros_logs"
storage:
"method": "filesystem"
"location": "default"
data: {
"ros_msg": "/rosout",
}
collection:
method: "event"
watch_topic: "/rosout"
If storage.method
is database ensure that your database is accessible at the host/port in the config file found at
storage.config
. To launch the default database roslaunch topic_store start_database.launch
, the default
database creates/uses a Mongo 4.2 server instance in the default ${HOME}/.ros/topic_store/database
folder exposed on localhost:65530
(defined in storage.config
of the scenario file).
Docker is a requirement to use a database backend to avoid conflicts with system mongo. A utility script is provided in
scripts/install_docker.sh
if you do not have it installed.
Launch your data collection scenario!
roslaunch topic_store run_scenario.launch scenario_file:="/path/to/your/scenario/file.yaml"
Examples
Database data
The below example shows how to load and use files stored in a database.
import topic_store as ts
# Read data
storage = ts.load("/path/to/scenario/file/containing/db/connection/info.yaml")
for item in storage:
print("As Python Dict", item.dict) # or item["key"]
print("As ROS Msgs", item.msgs) # or item("key")
# Write data
storage.insert_one({"important_data": "topic store is great!"})
Filesystem data
The below example shows how to load and use .topic_store
files, saved from when scenarios are ran with the
storage_method="filesystem"
option.
import topic_store as ts
# Read data
storage = ts.load("/path/to/file.topic_store")
for item in storage:
print("As Python Dict", item.dict) # or item["key"]
print("As ROS Msgs", item.msgs) # or item("key")
# Write data
storage.insert_one({"important_data": "topic store is great!"})
Launch a database
When launching a data collection scenario where storage.method==database
you must also launch the database or
ensure it's already running at the uri defined in the host/port parameters of the config file located at storage.config
.
To launch a database prior to running a data collection scenario.
roslaunch topic_store start_database.launch scenario_file:="/path/to/your/scenario/file.yaml"
Convert to ROS bags
Filesystem .topic_store
files and database collections can be converted to ROS bags.
# Convert filesystem files to rosbags
rosrun topic_store convert.py -i input.topic_store -o output.bag
# Convert database collections to rosbags
# Pass scenario file containing connection and collection info as the input
rosrun topic_store convert.py -i scenario_config.yaml -o output.bag
Convert between Filesystem and Database
Filesystem .topic_store
files can be migrated to a MongoDB database. Pass the scenario file that contains the database
connection information as the output file.
rosrun topic_store convert.py -i input.topic_store -o scenario_config.yaml
Database collections can be converted to a filesystem .topic_store
file. Pass the scenario file that contains the database
connection information as the input file.
rosrun topic_store convert.py -i scenario_config.yaml -o output.topic_store
Export from Mongodb query into rosbag/filesystem
Example call:
- With typical mongodb URI for SSL and authentication
- On database
ff_rasberry
and collection2020_riseholme_framos_cameras
- Specify database connection URI (
-i
) with db nameauthSource=database
- Including a query (
-q
) for a specific document - Add projection to return only sub-documents (
-p
)
convert.py -i "mongodb://USER:PASS@HOST:PORT/?authSource=ff_rasberry&tls=true&tlsAllowInvalidCertificates=true" -c 2020_riseholme_framos_cameras -q '{"_id":"ObjectId(5f115ee6af915351df739757)"}' -p '{"cameras.top.color":1, "robot": 1}' -o out.bag
Storing high bandwidth image data
To store a lot of image data you will be limited to the IO of your machine. Topic Compression offers a compression solution to increase capture performance.
# Install the ROS package
cd catkin_ws/src
git clone https://github.com/RaymondKirk/topic_compression
catkin build topic_compression
# Compression/Decompression is chosen automatically so just pass the input topic name and optionally the out topic name
rosrun topic_compression run in:=/camera/colour/image_raw # out=/camera/colour/image_raw/compressed
rosrun topic_compression run in:=/camera/depth/image_raw # out=/camera/depth/image_raw/compressed
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for topic_store-0.1.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ca317b8719cd4e80c6d5632fed8f05457e67d930daae72c7a22bf5ed7790cd75 |
|
MD5 | 3933939b1888c8dba9aeb257ccb0629e |
|
BLAKE2b-256 | 45be372310ad66fe28503f1291c566a6772dfadbdcc524eb1a55a72733ccded6 |
Hashes for topic_store-0.1.5-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | afd29751c079665bddeb9ad9d6decb8150cee190d10be36e68c7a2d58b664dfa |
|
MD5 | aab292dbb754ac7985e224caac316e32 |
|
BLAKE2b-256 | 2ed8957408e37c098ad6ce8ca7357f416e30991272d037825bad8e1fd4212859 |