Skip to main content

ALeRCE Alert Processing Framework.

Project description

Alert Processing Framework (apf)

Documentation Status codecov

apf is a framework developed to create a dockerized pipeline to process an alert stream, that can be easily be deployed in a local machine or distributed using Kubernetes.

First developed to process ZTF data it is capable to be used for any stream/static data processing pipeline.

Installing apf

apf installation can be done with pip

pip install apf_base

This will install the apf python package and apf command line script.

apf design

apf is based on steps conected through Apache Kafka topics.

Each step is composed by a consumer and is isolated from other steps inside a docker container.

When running, the step calls the execute() method for each message consumed. A step can have multiple producers and databases back-ends plugins that can be accessed inside the execute method to have a more complex logic.

This generic step greatly reduce the development of each component of the pipeline and make it easier to test each component separately.

apf features

  • Automatic Metric Sender (KafkaMetrics)
  • Automatic Code Generation (apf new-step <step_name>)
  • Multiple Consumer Plugins:
    • Kafka
    • AVRO
    • CSV
    • JSON
  • Producers:
    • Kafka
    • CSV
  • Metrics:
    • Kafka

apf Quickstart

A quick-start guide to create a new step can be found here.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

apf_base-1.0.8.tar.gz (15.1 kB view hashes)

Uploaded source

Built Distribution

apf_base-1.0.8-py3-none-any.whl (23.0 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page