Reusable TOMToolkit app for listening to kafka streams.
Project description
tom-alertstreams
tom-alertstreams
is a reusable TOM Toolkit app for listening to kafka streams.
tom-alertstreams
provides a management command, readstreams
. There are no urlpatterns
,
no Views, and no templates. The readstreams
management command reads the settings.py
ALERT_STREAMS
configuration and starts listening to each configured Kafka stream. It is not expected
to return, and is intended to run along side your TOM's server component. The ALERT_STREAMS
configuration (see below) tells readstreams
what streams to access, what topics to listen to,
and what to do with messages that arrive on a given topic.
Installation
-
Install the package into your TOM environment:
pip install tom-alertstreams
-
In your project
settings.py
, addtom_alertstreams
to yourINSTALLED_APPS
setting:INSTALLED_APPS = [ ... 'tom_alertstreams', ]
At this point you can verify the installation by running ./manage.py
to list the available
management commands and see
[tom_alertstreams]
readstreams
in the output.
Configuration
Each Kafka stream that your TOM listens to (via readstreams
) will have a configuration dictionary
in your settings.py
ALERT_STREAMS
. ALERT_STREAMS
is a list of configuration dictionaries, one
dictionary for each Kafka stream. Here's an example ALERT_STREAMS
configuration for two Kafka streams:
SCiMMA Hopskotch and
GCN Classic over Kafka.
ALERT_STREAMS = [
{
'ACTIVE': True,
'NAME': 'tom_alertstreams.alertstreams.hopskotch.HopskotchAlertStream',
'OPTIONS': {
'URL': 'kafka://kafka.scimma.org/',
'USERNAME': os.getenv('SCIMMA_AUTH_USERNAME', None),
'PASSWORD': os.getenv('SCIMMA_AUTH_PASSWORD', None),
'TOPIC_HANDLERS': {
'sys.heartbeat': 'tom_alertstreams.alertstreams.hopskotch.heartbeat_handler',
'tomtoolkit.test': 'tom_alertstreams.alertstreams.alertstream.alert_logger',
'hermes.test': 'tom_alertstreams.alertstreams.alertstream.alert_logger',
},
},
},
{
'ACTIVE': True,
'NAME': 'tom_alertstreams.alertstreams.gcn.GCNClassicAlertStream',
# The keys of the OPTIONS dictionary become (lower-case) properties of the AlertStream instance.
'OPTIONS': {
# see https://github.com/nasa-gcn/gcn-kafka-python#to-use for configuration details.
'GCN_CLASSIC_CLIENT_ID': os.getenv('GCN_CLASSIC_CLIENT_ID', None),
'GCN_CLASSIC_CLIENT_SECRET': os.getenv('GCN_CLASSIC_CLIENT_SECRET', None),
'DOMAIN': 'gcn.nasa.gov', # optional, defaults to 'gcn.nasa.gov'
'CONFIG': { # optional
# 'group.id': 'tom_alertstreams - llindstrom@lco.global',
# 'auto.offset.reset': 'earliest',
# 'enable.auto.commit': False
},
'TOPIC_HANDLERS': {
'gcn.classic.text.LVC_INITIAL': 'tom_alertstreams.alertstreams.alertstream.alert_logger',
'gcn.classic.text.LVC_PRELIMINARY': 'tom_alertstreams.alertstreams.alertstream.alert_logger',
'gcn.classic.text.LVC_RETRACTION': 'tom_alertstreams.alertstreams.alertstream.alert_logger',
},
},
}
]
The configuration dictionary for each AlertStream
subclass will contain these key-value pairs:
ACTIVE
: Boolean which tellsreadstreams
to access this stream. Should beTrue
, unless you want to keep a configuration dictionary, but ignore the stream.NAME
: The name of theAlertStream
subclass that implements the interface to this Kafka stream.tom_alertstreams
will provideAlertStream
subclasses for major astromical Kafka streams. See below for instructions on Subclassing theAlertStream
base class.OPTIONS
: A dictionary of key-value pairs specific to theAlertStream
subclass given byNAME
. The doc string forAlertStream
subclass should document what is expected. Typically, a URL, authentication information, and a dictionary,TOPIC_HANDLERS
, will be required. See "SubclassingAlertStream
" below.
Alert Handling
documentation coming.
Subclassing AlertStream
documentation coming.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for tom_alertstreams-0.4.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 91b79b497003f4d1720e0af6e92ff846763015db429feec3a975982a9c0e2c00 |
|
MD5 | adfdb15527dad8c1f7081361e563c970 |
|
BLAKE2b-256 | 6f8e9fb17440074a3d89f78ac86615c155d24975a5284327054bb91147743d68 |