ALeRCE Alert Processing Framework.
Project description
Alert Processing Framework (apf)
apf is a framework developed to create a dockerized pipeline to process an alert stream, that can be easily be deployed in a local machine or distributed using Kubernetes.
First developed to process ZTF data it is capable to be used for any stream/static data processing pipeline.
Installing apf
apf installation can be done with pip
pip install apf_base
This will install the apf python package and apf command line script.
apf design
apf is based on steps conected through Apache Kafka topics.
Each step is composed by a consumer and is isolated from other steps inside a docker container.
When running, the step calls the execute() method for each message consumed. A step can have multiple producers and databases back-ends plugins that can be accessed inside the execute method to have a more complex logic.
This generic step greatly reduce the development of each component of the pipeline and make it easier to test each component separately.
apf features
- Automatic Metric Sender (KafkaMetrics)
- Automatic Code Generation (
apf new-step <step_name>
) - Multiple Consumer Plugins:
- Kafka
- AVRO
- CSV
- JSON
- Producers:
- Kafka
- CSV
- Metrics:
- Kafka
apf Quickstart
A quick-start guide to create a new step can be found here.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for apf_base-1.0.1.linux-x86_64.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | d15c721913ee889c70cf31d0c6e34c7e4d18795e90a112403f42f6a03aa211d4 |
|
MD5 | 127c56a41fc647342b90ee589cf52c16 |
|
BLAKE2b-256 | 3e9e0d93c65d6a5b31214bf3b785ad6a179aec7ac9e8c5bc408ee52131428cc1 |