astro-colibri-circular-parser

AI-assisted parsing of GCN circulars into structured, machine-readable multi-wavelength follow-up observations.

This is the extraction pipeline behind the GRB optical-afterglow feature of the Astro-COLIBRI multi-messenger astronomy platform, published as a standalone library. Given a circular (by number, URL, or raw text) it extracts reported observations — detections, upper limits, non-detections, X-ray/radio fluxes, redshifts — into a JSON record, with times normalised to seconds since trigger and optical photometry converted to a common observed-frame R_c-equivalent AB magnitude. As with any automated extraction, completeness and scientific values should be checked against the original circular before publication.

Associated manuscript: F. Schüssler et al., “AI-Assisted Extraction of Follow-up Observations from GCN Circulars in Astro-COLIBRI” (prepared for PASP; publication link to be added when available). Live service: the parsed results power the afterglow light curves at https://astro-colibri.com (documentation: https://astro-colibri.science/followupdoc).

How it works

circular number / URL ──► fetch (gcn.nasa.gov JSON archive)
                              │
                              ▼
              deterministic regex pre-analysis
        (source names, photometry hints, contact emails)
                              │
                              ▼
          event resolution (optional, pluggable)
     Astro-COLIBRI public API: trigger time, position, E(B-V)
                              │
                              ▼
         LLM structured extraction (OpenAI Responses API)
   strict JSON schema; the regex hints are advisory input only
                              │
                              ▼
                 photometric enrichment
   filter normalisation → common Rc band (AB), Galactic-extinction
   handling, absolute/relative time reconciliation vs. trigger
                              │
                              ▼
        follow-up payload + consistency checks (JSON)

Key design points:

• Structured outputs, not free text. The LLM must return JSON conforming to a strict schema (PHOTOMETRY_RESPONSE_SCHEMA); every object is closed (additionalProperties: false) and fully required, so missing values are explicit nulls.
• Deterministic scaffolding around the LLM. Callers can use the exported regex helpers as cheap prefilters; parse_circular(...) itself always runs the configured extraction provider. Regex hints point the model at magnitudes, limits and redshifts, but the prompt requires the model to verify them against the circular. Consistency checks flag extractions that need human review (magnitudes without filters, unparseable times, negative times-since-trigger, ...).
• Physics in code, not in the LLM. All photometric conversions (filter → common R_c band, Vega↔AB, extinction re-reddening, time arithmetic) are classical Python (circular_parser/photometry.py), unit-tested and independent of the model.
• No hidden state. The pipeline returns a plain dict; it performs no database writes and sends no notifications.

Install

git clone https://github.com/astro-transients/astro_colibri_circular_parser
cd astro_colibri_circular_parser
pip install .                # library + CLI
pip install ".[examples]"    # + notebook/plotting extras

Python ≥ 3.9. Dependencies: openai, requests, python-dateutil, and python-dotenv (for CLI .env loading).

Configure

Copy .env.example to .env and set your OpenAI API key (openAI_key or OPENAI_API_KEY). Never commit the key. All other settings are optional; see the table in .env.example and circular_parser/settings.py. A successful parse normally uses one LLM request. Transient failures can trigger the configured retries, and semantic time validation can invoke an optional fallback model. Provider billing therefore depends on the selected model and number of attempts.

Quickstart

from circular_parser import parse_circular

result = parse_circular(45049)   # fetches https://gcn.nasa.gov/circulars/45049

print(result["source_name"])                 # EP260626a
print(result["consistency_issues"])          # []
for obs in result["payload"]["observations"]:
    print(obs["observation_type"], obs["filter_raw"], obs["mag_raw"],
          obs["time_since_trigger_s"], obs["corrected"]["mag_rc_ab_gal"])

Command line:

python -m circular_parser 45049 --pretty                 # full result to stdout
python -m circular_parser 45049 --output result.json     # write to a file
python -m circular_parser 45049 --no-event-lookup        # skip event linking
python -m circular_parser 45049 --offline examples/cached/extraction_45049.json  # replays the LLM result

Cached extraction replay

The example notebook and the --offline CLI flag replay a cached extraction (examples/cached/extraction_45049.json) without an OpenAI API key. Enrichment, payload construction and consistency checks are identical to a live run.

The CLI flag replaces only the LLM request: when given a circular number or URL, the CLI still retrieves the circular from GCN and performs event lookup unless --no-event-lookup is also supplied. For a completely network-free run, use the Python API with cached circular content, a replay provider, and either a cached event= dictionary or resolve_events=False. The example notebook and the test suite demonstrate this pattern; pytest tests/ needs no network or credentials.

Human validation

Human review of stored reports belongs to the surrounding Astro-COLIBRI platform rather than this standalone parser. The parser returns consistency_issues and preserves the original circular text and provenance so callers can implement their own review workflow. The live service is described in the Astro-COLIBRI follow-up documentation.

Output data model

parse_circular(...) returns:

Each observation in payload["observations"] carries the reported values (filter_raw, mag_raw, mag_err, flux/flux_unit, time_raw, ...) plus derived quantities: time_since_trigger_s, is_upper_limit, and corrected (mag_ab, mag_rc_ab_gal — the observed-frame R_c-equivalent AB magnitude used for light curves — beta, correction_status).

Event linking (Astro-COLIBRI)

To convert absolute observation times to times since trigger — and to attach positions/extinction — the pipeline can resolve the circular against the public, unauthenticated Astro-COLIBRI API (/event, /source_details). This is optional: pass resolve_events=False to disable event lookup (relative times like "26.5 hours after the trigger" still work), supply your own event= dict, or plug in any other backend via EventLookup (three callables). Avoiding all network access also requires passing circular content directly instead of a circular number or URL. See circular_parser/events.py.

Examples

examples/parse_circular_demo.ipynb walks through: fetching a real circular, the regex pre-analysis, an offline replay of a cached extraction (renders fully without an API key), an optional live LLM extraction, and a light-curve plot of the enriched photometry.

Tests

pip install ".[tests]"
pytest tests/

The test suite uses cached data and test doubles; no network access or API keys are required.

License and citation

BSD 3-Clause (see LICENSE). If you use this code in a publication, please use the metadata in CITATION.cff to cite the associated manuscript (the final publication reference will be added when available) and cite the Astro-COLIBRI platform (Reichherzer et al. 2021, ApJS 256, 5).