isis-streaming-data-types 0.1.1

Python utilities for handling ISIS streamed data

Streaming Data Types

FlatBuffers is the format chosen for the ISIS data streaming system, derived from the ESS messaging system.

Schema ids

ID	File name	Description
f144	f144_logdata.fbs	Controls related log data, typically from EPICS or NICOS. Note: not to be used for array data
ev44	ev44_events.fbs	Multi-institution neutron event data for both single and multiple pulses
df12	df12_det_spec_map.fbs	Detector-spectrum map for Mantid
se00	se00_data.fbs	Used for storing arrays with optional timestamps, for example waveform data. Replaces senv.
ad00	ad00_area_detector_array.fbs	EPICS area detector array data
hs01	hs01_event_histogram.fbs	Event histogram stored in n dim array
ep01	ep01_epics_connection.fbs	Status or event of EPICS connection. Replaces ep00
json	json_json.fbs	Carries a JSON payload
pl72	pl72_run_start.fbs	File writing, run start message for file writer and Mantid
6s4t	6s4t_run_stop.fbs	File writing, run stop message for file writer and Mantid
answ	answ_action_response.fbs	Holds the result of a command to the filewriter
wrdn	wrdn_finished_writing.fbs	Message from the filewriter when it is done writing a file
x5f2	x5f2_status.fbs	Status update and heartbeat message for any software
fc00	fc00_forwarder_config.fbs	Configuration update for Forwarder
al00	al00_alarm.fbs	Generic alarm schema for EPICS, NICOS, etc.
da00	da00_dataarray.fbs	Pseudo-scipp DataArray with time-dependent and constant Variables
un00	un00_units.fbs	Engineering units update

Adding new schemas

Add .fbs file to `schemas directory

Check ess-streaming-data-types first; attempt not to diverge without reason. If a new schema is really needed, then add a new .fbs schema in the schemas directory.

Note: to generate code from .fbs schemas, you will need the flatc tool installed. It can be acquired from https://github.com/google/flatbuffers/releases .

Adding Python bindings

Python bindings have low-level code (autogenerated by flatc) in the fbschemas directory, but also manually-written convenience serializers and deserializers in the top-level of the python module.

When adding or modifying a schema:

• The low-level code must be generated by manually calling flatc --python schemas\some_schema.fbs and adding the resulting generated code to python/src/streaming_data_types/fbschemas.
• Manually-written serializers & deserializers will need to be updated, and added to the lists in __init__.py.

Rust bindings

Rust bindings have low-level code in flatbuffers_generated, and a small high-level wrapper to deserialize any message.

When adding or modifying a schema:

• The low-level code must be generated by calling generate_rust_bindings.py
• The wrapper (defined in lib.rs) will need to be updated with the new schema.