Fink Alert Broker Module for the TOM Toolkit

This repository hosts the Fink Alert Broker Module for the TOM Toolkit. Fink is a broker currently connected to ZTF, and that will handle the massive alert stream from LSST. More information on Fink at https://fink-broker.org/. You can easily install the module using pip:

pip install tom-fink

Then add the tom_fink app to your list of INSTALLED_APPS in settings.py:

INSTALLED_APPS = [
    ...
    'custom_code',
    'tom_fink'
]

This should give you access to the Dataservice UI.

Polling data from the REST API

As of tom-fink version 0.4, the module simply uses the Fink REST API to retrieve alerts. Note that the Fink databases are updated once a day with the previous night alert data (hence you do not get live alerts for the moment). Users can perform:

• Search by ZTF object ID
• Cone Search
• Search by derived alert class
• Search by Solar System name (currently disabled)

Polling data from the Fink livestream service

Alternatively from tom_fink version 0.6.0, users can poll Fink substreams in real-time, using the livestream service. First you need to add tom_alertstreams and tom_fink to your INSTALLED_APPS setting in your project settings.py:

INSTALLED_APPS = [
    ...
    'custom_code',
    'tom_alertstreams',
    'tom_fink'
]

Then add the following block to your settings.py:

ALERT_STREAMS = [
    {
        'ACTIVE': True,
        'NAME': 'tom_fink.alertstream.FinkAlertStream',
        'OPTIONS': {
            'URL': os.getenv('FINK_CREDENTIAL_URL', 'set FINK_CREDENTIAL_URL value in environment'),
            'USERNAME': os.getenv('FINK_CREDENTIAL_USERNAME', 'set FINK_CREDENTIAL_USERNAME value in environment'),
            'GROUP_ID': os.getenv('FINK_CREDENTIAL_GROUP_ID', 'set FINK_CREDENTIAL_GROUP_ID value in environment'),
            'TOPIC': os.getenv('FINK_TOPIC', 'set FINK_TOPIC value in environment'),
            'MAX_POLL_NUMBER': os.getenv("FINK_MAX_POLL_NUMBER", 1e10),
            'TIMEOUT': os.getenv('FINK_TIMEOUT', 10),
            'TOPIC_HANDLERS': {
                'fink.stream': 'tom_fink.alertstream.alert_logger',
            },
        },
    },
]

and declare the following variables in your ~/.bash_profile:

export FINK_CREDENTIAL_USERNAME= # provided when registering to Fink livestream
export FINK_CREDENTIAL_GROUP_ID= # provided when registering to Fink livestream
export FINK_CREDENTIAL_URL= # provided when registering to Fink livestream
export FINK_TOPIC= # str, topic name to poll
export FINK_MAX_POLL_NUMBER= # int, number maximum of alerts to poll before closing the connection. Default is 1e10 (never stops).
export FINK_TIMEOUT= # int, in seconds. Default is 10 seconds if not set

Credentials are provided when registering to Fink livestream (see https://fink-broker.readthedocs.io/en/latest/services/livestream). The available topic names and description can be found at https://fink-broker.readthedocs.io/en/latest/services/livestream. Note that we currently allow to poll only one topic at once (multi-topic is planned for later). Then launch the readstreams service:

./manage.py readstreams

You should see a log similar to:

registering new views: args: ('groups', <class 'tom_common.api_views.GroupViewSet'>, 'groups'), kwargs: {}
registering new views: args: ('targets', <class 'tom_targets.api_views.TargetViewSet'>, 'targets'), kwargs: {}
registering new views: args: ('targetextra', <class 'tom_targets.api_views.TargetExtraViewSet'>, 'targetextra'), kwargs: {}
registering new views: args: ('targetname', <class 'tom_targets.api_views.TargetNameViewSet'>, 'targetname'), kwargs: {}
registering new views: args: ('targetlist', <class 'tom_targets.api_views.TargetListViewSet'>, 'targetlist'), kwargs: {}
registering new views: args: ('observations', <class 'tom_observations.api_views.ObservationRecordViewSet'>, 'observations'), kwargs: {}
registering new views: args: ('dataproducts', <class 'tom_dataproducts.api_views.DataProductViewSet'>, 'dataproducts'), kwargs: {}
registering new views: args: ('reduceddatums', <class 'tom_dataproducts.api_views.ReducedDatumViewSet'>, 'reduceddatums'), kwargs: {}
NumExpr defaulting to 8 threads.
FinkAlertStream.listen opening stream: IP:PORT with group_id: julien6 (call number: 0)
read_streams FinkAlertStream TID=378166 ; thread identifier=125271444649536
readstreams Command.handle() returning...
fink.alert_logger topic: fink_sso_ztf_candidates_ztf
fink.alert_logger value: ZTF24aakwfsz emitted 2024-04-23 07:26:57.998412 (received 2024-04-29 10:45:01)
FinkAlertStream.listen opening stream: IP:PORT with group_id: julien6 (call number: 1)
fink.alert_logger topic: fink_sso_ztf_candidates_ztf
fink.alert_logger value: ZTF24aakwftv emitted 2024-04-23 07:27:39.003823 (received 2024-04-29 10:45:01)
...

Each new alert will be saved as a Target. If more than one alert with the name objectId are emitted, you will be warned:

fink.alert_logger value: ZTF24aakwfsu emitted 2024-04-23 07:26:57.998412 (received 2024-04-29 08:18:07)
Target ZTF24aakwfsu already in the database

and the program will continue. Probably we should group these in the future, instead of ignoring subsequent alerts. Then launch the app (do not close the previous process!):

./manage.py runserver

and you should have access to your targets, that are accessible from their topic name:

Testing & debugging the connection

Before running in production, we advise to make tests using a test stream, and polling a few alerts:

export FINK_CREDENTIAL_USERNAME= # provided when registering to Fink livestream
export FINK_CREDENTIAL_GROUP_ID= # provided when registering to Fink livestream
export FINK_CREDENTIAL_URL= # provided when registering to Fink livestream
export FINK_TOPIC=fink_sso_ztf_candidates_ztf
export FINK_MAX_POLL_NUMBER=2
export FINK_TIMEOUT=10

This way, you will poll alerts two-by-two, and you will have a reservoir of thousands of alerts (solar system objects)!

Running the tests for the app

In order to run the tests, run the following in your virtualenv:

python tom_fink/tests/run_tests.py

Todo list

• Add a test suite (preferably running on GitHub Actions)
• Add a linter (preferably running on GitHub Actions)
• Update the Query Form with all API features
• Enable querying live streams using the Fink Kafka client
• Update visualisation of targets from the livestream service