Alert Processing Framework (apf)
apf is a framework developed to create a dockerized pipeline to process an alert stream, that can be easily be deployed in a local machine or distributed using Kubernetes.
First developed to process ZTF data it is capable to be used for any stream/static data processing pipeline.
Installing apf
apf installation can be done with pip
pip install apf_base
This will install the apf python package and apf command line script.
apf design
apf is based on steps conected through Apache Kafka topics.
Each step is composed by a consumer and is isolated from other
steps inside a docker container.
When running, the step calls the execute() method for each message or message batch consumed. A step can have multiple producers and databases back-ends plugins that can be accessed inside the execute method to have a more complex logic.
This generic step greatly reduce the development of each component of the pipeline and make it easier to test each component separately.
apf features
- Automatic Metric Sender (KafkaMetrics)
- Automatic Code Generation (
apf new-step <step_name>) - Multiple Consumer Plugins:
- Kafka
- AVRO
- CSV
- JSON
- Producers:
- Kafka
- CSV
- Metrics:
- Kafka
apf Quickstart
A quick-start guide to create a new step can be found here.
1 maintainer
Release files for apf-base 2.5.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| apf_base-2.5.2.tar.gz | 18.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| apf_base-2.5.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 46.0 kB
Release files / apf_base-2.5.2.tar.gz
| Download URL | apf_base-2.5.2.tar.gz |
|---|---|
| Size | 18.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c60b80db5793c6ee5499ac5286aaa734221e187813f6c37eb184ebd762685fde
|
|
BLAKE2b-256 checksum How to use checksums |
a10c505c0bd9651fffd695649978f6ef5efe026e982c268662aadda5ba8dc299
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.4
|
Release files / apf_base-2.5.2-py3-none-any.whl
| Download URL | apf_base-2.5.2-py3-none-any.whl |
|---|---|
| Size | 27.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
eb495838a5159fae030a514f901e4fe028e58c058dcf95bf0b9fc2c1a208343f
|
|
BLAKE2b-256 checksum How to use checksums |
c138feaf618aad398d8d1ca5d5eb8985bcfc35bc7448fb8c5daab4a3fda00a02
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.4
|
