dq-genesis

Pipeline construction and execution for the creation of the data quality vector.

dq-genesis parses a YAML DQ flag config, assembles an sgn.Pipeline from the evaluators and transforms provided by sgn-dq, and runs it against frame data (offline frame cache), shared-memory frames (online), or an Arrakis stream. The output is a uint32 DQG-DQ_VECTOR channel written to a GWF frame stream.

Install

pip install dq-genesis                   # from PyPI
pip install -e /path/to/dq-genesis       # editable install from a source checkout

dq-genesis depends on sgn-dq (evaluators, Not, Parity), sgn-ts, sgn-ligo, and sgn.

Quick start

# Validate a config
dq-genesis --flags configs/dqmodule_h1.yaml --dry-run

# Offline run against a frame cache
dq-genesis --flags configs/dqmodule_h1.yaml \
  --gps-start 1234567890 --gps-end 1234568000 \
  --frame-cache h1.cache --output-dir /tmp/out

# Bounded replay from Arrakis (omit --gps-start/--gps-end for a live stream)
dq-genesis --flags configs/dqmodule_h1.yaml \
  --arrakis --gps-start 1234567890 --gps-end 1234568000 \
  --output-dir /tmp/out

See src/dqgenesis/cli.py --help for the full argument surface.

Output destinations

The assembled DQ vector can be fanned out to one or more sinks. Select any combination of the --publish-* flags; if none is given it defaults to disk, preserving the original frame-writing behaviour.

• --publish-disk — write the DQ vector to GWF frame files. Control the destination with --output-dir (and --write-path, --frame-duration).
• --publish-arrakis — publish the DQ vector channel back to an Arrakis server. Requires --publisher-id; pass --replay-id to publish into a replay namespace.

# Save to GWF frames (default)
dq-genesis --flags configs/dqmodule_h1.yaml \
  --gps-start 1234567890 --gps-end 1234568000 \
  --frame-cache h1.cache --publish-disk --output-dir /tmp/out

# Publish back into Arrakis
dq-genesis --flags configs/dqmodule_h1.yaml \
  --arrakis --gps-start 1234567890 --gps-end 1234568000 \
  --publish-arrakis --publisher-id <publisher_id>

# Both at once — the DQ vector is fanned out to disk and Arrakis
dq-genesis --flags configs/dqmodule_h1.yaml \
  --arrakis --gps-start 1234567890 --gps-end 1234568000 \
  --publish-disk --output-dir /tmp/out \
  --publish-arrakis --publisher-id <publisher_id>

The output channel is named <IFO>:DQG-DQ_VECTOR by default; override it with --output-channel.

Monitoring

Pass --monitor to serve live monitoring for the running pipeline via sgnmon:

dq-genesis --flags configs/dqmodule_h1.yaml --arrakis --monitor

This starts a background web server (default port 9090; --monitor-port changes it and implies --monitor, with 0 picking a free port) exposing:

• / -- a live dashboard drawing the pipeline graph with per-element rates, latencies, and gap fractions
• /metrics -- Prometheus metrics for scraping, including the DQ vector itself: dqgenesis_flag_active and dqgenesis_flag_active_fraction (each flag's state in the latest sample and the fraction of the latest frame it was set, labelled bit and flag), dqgenesis_flag_transitions_total (how often each flag has flipped), dqgenesis_dq_vector (the latest value) and dqgenesis_dq_vector_gps_seconds, dqgenesis_dq_vector_samples_total, and dqgenesis_parity_violations_total. All are labelled ifo.
• /health -- a JSON health report (HTTP 503 when unhealthy), usable directly by container orchestration or sgnmon check; the DQ vector's own checks are also served one at a time at /health/dq_vector_fresh and /health/dq_parity, for monitors that only see status codes
• /readyz and /healthz -- readiness (the pipeline is running and every check passes) and liveness (the run loop strided recently) from the pipeline's own lifecycle
• /status -- a JSON snapshot of all probes

The data-freshness check on every pad tolerates three missed --frame-durations (30 s at least). dq_vector_fresh fails when no DQ vector sample has been produced for five frame durations (5 minutes at least), measured in wall-clock time from startup or the last sample: inputs can keep flowing, so every pad looks healthy, while the bit vector stalls waiting on one evaluator. dq_parity fails once any output sample has failed odd parity (it always passes when the config has no parity bit).

Under --monitor the DQ vector is also fed to a small observer sink, dq_monitor, next to the frame and Arrakis sinks; without it the pipeline graph is exactly as before.

The lifecycle also reaches systemd: under a Type=notify unit (or a Podman quadlet with Notify=true) the process sends READY=1, WATCHDOG=1 as it progresses and STOPPING=1 on exit, so WatchdogSec= restarts a hung process; the unit's timeout must exceed the liveness threshold above.