Pastro

Pastro (distributed as sgnl-pastro; the Python import package is still pastro) is a package to estimate the probabilities of astrophysical origins of the gravitational wave candidates detected by the GstLAL/SGNL pipeline, for the purpose of assisting in real-time multi-messenger follow-up observations. This repository develops and operates our p_astro model for the evaluation of p(category|data) with BNS, BBH, NSBH, and Terrestrial (noise) events.

For more detailed information on the operation manual of pastro, please find the information on gwsci.org: https://gwsci.org/ops/pastro

Install & Setup

All dependencies are pip-installable, so a plain pip install of the checkout is sufficient on any machine (laptop, ICDS, CIT, ...):

$ git clone https://git.ligo.org/greg/sgnl-pastro.git
$ cd sgnl-pastro
$ pip3 install .

If you plan to git pull updates, install in editable mode instead with pip3 install -e . — then the installed package tracks your checkout and you do not need to reinstall after every git pull. For development, see the Development section below. The package can also be installed the same way inside a GstLAL/SGNL singularity container if you want it co-located with a running analysis, but no container is required.

Installing with pip also installs all of the command-line tools (e.g. sgnl-pastro-initialize-fgmc-model, sgnl-pastro-calculate-fgmc, ...) as console scripts on your PATH. All commands are namespaced under the sgnl-pastro- prefix; run sgnl-pastro-<TAB><TAB> to list them.

Notice: the package no longer imports gstlal or manifold — those dependencies were ported to strike/igwn_ligolw (and cosmology_utils was vendored into pastro.cosmology_utils). Some tools do, however, read ranking-statistic PDF files that may be produced by either GstLAL or the sgnl/strike stack: sgnl-pastro-offline and the pastro.pastro model methods load them through pastro.io.load_rankstatpdf, which auto-detects the producer by default. You can force the format with the rankstat_format config key for sgnl-pastro-offline (or the fmt argument of pastro.io.load_rankstatpdf). GstLAL is not required to be installed to read GstLAL-format files; only the input file itself is needed.

FGMC method (O4 low-latency)

Ray, Anarya. et. al. (2023), dcc-link: https://dcc.ligo.org/P2300141

This section walks through the full preparation of the online (low-latency) FGMC p_astro, from raw inputs to a deployable model file, with working commands. Three input artifacts feed the model build:

Input	Produced by	Documented in
rates & sensitive volumes (rates_V.json)	sgnl-pastro-calculate-rates	Rates_for_O4.md
template-weight files (ceff*.h5, one per category)	manifold (external)	template-weights-fgmc-using-manifold.md
ranking-statistic PDF (*DIST_STAT_PDFS*.xml.gz)	the running analysis	—

If you just want to see the whole chain run right now, with no cluster access or real analysis products, skip to Step 6: runnable quickstart — examples/fgmc/online/ generates synthetic stand-ins for all of the inputs and runs Steps 4–5 verbatim.

Step 1: estimate rates and sensitive volumes

The expected astrophysical rates and the sensitive volumes scaled to the new run are computed by sgnl-pastro-calculate-rates from the previous runs' FGMC rate estimates, BNS ranges, run times, and recovered hypervolumes. The full procedure and the provenance of every number is documented in Rates_for_O4.md; the O4 invocation is:

sgnl-pastro-calculate-rates \
    --rates-astro '{"BBH": 40.0, "NSBH": 68.0, "BNS": 163.0}' \
    --VT-old '{"BBH": 1.63, "NSBH": 0.048, "BNS": 0.014}' \
    --dbns-old '{"O1": 0.08, "O2": 0.1, "O3": 0.12}' \
    --runtime-old '{"O1": 0.333333333333, "O2": 0.75, "O3": 1.0}' \
    --dbns-new 0.160 \
    --threshold-old 8760.12 \
    --terrestrial-old 11541 \
    --T-inj-start 1262304018 --T-inj-stop 1265760018 \
    --N-inj '{"BBH": 118036.67, "BNS": 118036.67, "NSBH": 118036.67}' \
    --z-max '{"BBH": 1.9, "BNS": 0.15, "NSBH": 0.25}' \
    --output rates_V_O4.json

The output JSON holds three dictionaries: rates_astrophysical (→ --rates in Step 4), rates_injections (→ --rates-inj; deprecated for O4, see the note in Rates_for_O4.md) and V (→ --V-new). The --T-inj-*/--N-inj/--z-max options only affect the deprecated injection rates.

Step 2: build the template weights

The per-category template weights encode p(template, SNR | category) and are produced from the mass model (with NS-boundary mass cuts) by the external manifold tool — condor DAG steps documented in template-weights-fgmc-using-manifold.md.

Naming requirement: the model loader globs ceff*.h5 in --weights-dir and identifies each file's category by substring match on the filename (e.g. ceff_BBH.h5 → BBH). Manifold's outputs are named like H1L1V1-O4_MANIFOLD_BANK_SALPETER_BBH_-0-2000000000.h5, so rename (or symlink) them to ceff_<CATEGORY>.h5 when assembling the weights directory. Each file must contain the coefficients, SNR and template_id datasets.

Step 3: obtain a ranking-statistic PDF

The likelihood-ratio normalization constant A is computed from the analysis' ranking-statistic PDF file (e.g. H1L1V1-GSTLAL_DIST_STAT_PDFS-0-0.xml.gz from the online run's dist_stat_pdfs directory). Both GstLAL-format and sgnl/strike-format documents are read; the producer is auto-detected (see the format notice in the install section above). This input is technically optional (A defaults to 1.0 without it) but required for a correctly normalized production model.

Step 4: build the model file

With the three inputs in hand, build the model .h5 (values below are the O3-era production numbers from examples/fgmc/test_model/Makefile; the FAR threshold is in expected counts per Julian year — 730.5 is 2 per day):

sgnl-pastro-initialize-fgmc-model \
    --rankstat-filename /path/to/H1L1V1-GSTLAL_DIST_STAT_PDFS-0-0.xml.gz \
    --weights-dir /path/to/template-weights \
    --rates '{"BBH": 18.0730098234, "NSBH": 52.9087878433, "BNS": 662.448825556}' \
    --rates-inj '{"BBH": 4045.100324417181, "BNS": 1086074.923130263, "NSBH": 269801.54621791927}' \
    --V-new '{"BBH": 3.720777380922114, "NSBH": 0.13036984478340222, "BNS": 0.01447408360378373}' \
    --far-threshold 730.5 \
    --output fgmc_model.h5

The --rates/--rates-inj/--V-new dictionaries must all cover the same category set. Optionally pass --template-database <bank.sqlite> to take the template-id mapping from a template-bank database instead of the weight files.

Step 5: validate the model and evaluate events

Evaluate a single trigger from hand-entered numbers (template id, network SNR, ln likelihood ratio):

sgnl-pastro-calculate-fgmc --fgmc-model fgmc_model.h5 --templateid 7012 --snr 14.487 --lnlr 52.619
{"BBH": 0., "BNS": 0.9999929320428521, "NSBH": 4.1997623780371124e-16, "Terrestrial": 0.}

Evaluate LIGO_LW coinc documents (e.g. the coinc.xml files the online pipeline uploads to GraceDB) with sgnl-pastro-calculate-fgmc-coinc:

sgnl-pastro-calculate-fgmc-coinc --fgmc-model fgmc_model.h5 coinc1.xml coinc2.xml.gz ...

For every coincident event in every input file this prints one JSON object per line containing the trigger data (end time, combined FAR, SNR, ln LR, template id) and the evaluated p_astro. The trigger inputs are read from the standard tables: likelihood from coinc_event, network snr/end_time/combined_far from coinc_inspiral, and the template id from the Gamma0 column of the first linked sngl_inspiral row (override with --template-id-column if your documents store it elsewhere). Pass --far-threshold <Hz> to skip events whose combined_far is above the given rate, mirroring the online FAR cut.

The same tool evaluates any p_astro model file with --model in place of --fgmc-model: the numbered models of this package, or an sgn-manifold combined model (sgn-manifold-cbc-p-astro-combine, stamped with its runtime normalization by sgn-manifold-cbc-p-astro-configure; needs pip install sgnl-pastro[manifold]). That model classifies each event from its template id and combined_far alone (likelihood/snr are ignored), returning the same {BNS, NSBH, BBH, Terrestrial} JSON, so the two models can be compared event by event on the same documents:

sgnl-pastro-calculate-fgmc-coinc --model pastro-model.h5 coinc1.xml coinc2.xml.gz ...

pastro.load(filename) performs the same dispatch for programmatic use (SGNL's online uploader loads models this way).

To compare against a GraceDB MDC injection campaign (requires a valid X.509 proxy and network access to GraceDB):

sgnl-pastro-calculate-fgmc-test-inj --output-file pastro_inj.txt --fgmc-model fgmc_model.h5 --injection-file injections.xml.gz

Production-style Makefiles with the real cluster paths for Steps 4–5 live in examples/fgmc/Makefile and examples/fgmc/test_model/Makefile.

Step 6: runnable quickstart (no cluster access needed)

examples/fgmc/online/ contains a self-contained version of Steps 4–5: make_demo_inputs.py generates synthetic template weights (ceff_*.h5) and a two-event coinc.xml.gz, and the Makefile builds a model from them and evaluates both a hand-entered trigger and the coinc document. The test suite runs this example end-to-end (tests/test_example_fgmc_online.py), so it is guaranteed to work:

$ cd examples/fgmc/online
$ make
...
cat single_trigger.json
{"BBH": 0.016503467977679633, "BNS": 0.9493390523474371, "NSBH": 0.03400160750475215, "Terrestrial": 0.00015587217013113275}
...
cat coinc_results.jsonl
{"template_id": 100.0, "likelihood": 12.0, "snr": 9.0, "end_time": 1370000000.25, "combined_far": 1e-10, "file": "coinc.xml.gz", "p_astro": {"BBH": 0.016503467977679633, "BNS": 0.9493390523474371, "NSBH": 0.03400160750475215, "Terrestrial": 0.00015587217013113275}}
{"template_id": 300.0, "likelihood": 1.5, "snr": 6.5, "end_time": 1370000100.0, "combined_far": 1e-05, "file": "coinc.xml.gz", "p_astro": {"BBH": 0.5512015358585471, "BNS": 9.699276100104307e-06, "NSBH": 0.00038095949183503054, "Terrestrial": 0.44840780537351776}}

The loud, low-FAR event in the BNS-weighted template comes out confidently astrophysical (and identical to the hand-entered evaluation of the same numbers — the two tools share the model math); the marginal, high-FAR event splits between BBH and Terrestrial. The demo model omits --rankstat-filename (no analysis behind it, so A = 1); everything else is exactly the production path.

Step 7: deploy for online running

Handing the model file to the running low-latency analysis is an operations procedure, documented at https://gwsci.org/ops/pastro.

Development

For development, install the package in editable mode together with the test and lint dependencies:

$ pip3 install -e ".[dev]"

The top-level Makefile provides the standard development targets:

$ make format      # run isort and black
$ make lint        # run flake8
$ make type-check  # run mypy
$ make test        # run pytest
$ make all         # run all of the above