arrakis-lldd-connector

A service built on SGN, connecting data between LLDD, Arrakis, and frame files.

Stream gravitational-wave detector timeseries between the low-latency data distribution system (LLDD), the Arrakis timeseries service, and GWF frame files. Pick a --source and a --sink and the connector builds and runs the SGN pipeline between them.

Resources

• Source Code
• Issue Tracker

Installation

pip install arrakis-lldd-connector

Or from source:

git clone git@git.ligo.org:ngdd/arrakis-lldd-connector.git
cd arrakis-lldd-connector
pip install .

Features

• Sources: LLDD (Kafka frame streams), Arrakis, GWF frame file directories, and synthetic test signals
• Sinks: Arrakis, LLDD, GWF frame files, and a null sink for testing
• Any source can be paired with any sink
• Live streams are resilient to interruptions: sources reconnect with capped exponential backoff and bridge outages with gap buffers
• Bounded (--start/--end) or continuous live operation
• Arrakis replay namespace support for both streaming and publishing (--replay-id)
• Frame writing options: file retention (--max-files, --retention-time), and skipping files that contain no real data (--skip-all-gap)

Quickstart

The CLI shows contextual options: pass --source and/or --sink with --help to see the options relevant to that combination, e.g.

arrakis-lldd-connector --source frames --sink arrakis --help

Publish live DAQ data from LLDD to Arrakis

arrakis-lldd-connector --source lldd --sink arrakis \
    --ifo L1 --topic Live_LLO_Data \
    --bootstrap-servers kafka1:9092,kafka2:9092 \
    --arrakis-url grpc://arrakis-online1:31206 \
    --publisher-id L1-lldd

The channels to publish are discovered from the publisher's registration on the Arrakis server.

Write an Arrakis stream to frame files

The Arrakis source reads the server URL from the ARRAKIS_SERVER environment variable:

export ARRAKIS_SERVER=grpc://arrakis-online1:31206
arrakis-lldd-connector --source arrakis --sink frames \
    --channels L1:GDS-CALIB_STRAIN L1:GDS-CALIB_STATE_VECTOR \
    --frame-duration 64 --output-dir /data/frames \
    --skip-all-gap

Without --start and --end this streams live data continuously; press Ctrl+C to stop.

Publish frame files to LLDD

Watch a directory for new frame files and publish them to Kafka:

arrakis-lldd-connector --source frames --sink lldd \
    --channels H1:GDS-CALIB_STRAIN \
    --watch-dir /data/live/H1 \
    --ifo H1 --topic Live_LHO_Data \
    --bootstrap-servers kafka1:9092

Test a pipeline with synthetic data

arrakis-lldd-connector --source fake --sink null \
    --channels H1:TEST --rate 2048 --signal-type sin \
    --start 0 --end 10

Usage with Docker

The container image is hosted on containers.ligo.org. Pull the latest version with:

podman pull docker://containers.ligo.org/ngdd/arrakis-lldd-connector:latest

Run with podman (or docker):

podman run --rm --net=host \
    docker://containers.ligo.org/ngdd/arrakis-lldd-connector:latest \
    --source lldd --sink arrakis \
    --ifo L1 \
    --arrakis-url grpc://arrakis-online1:31206 \
    --publisher-id L1-lldd \
    --bootstrap-servers kafka1:9092,kafka2:9092 \
    --topic Live_LLO_Data