Skip to main content

ALeRCE Alert Processing Framework.

Project description

Alert Processing Framework (apf)

Documentation Status codecov

apf is a framework developed to create a dockerized pipeline to process an alert stream, that can be easily be deployed in a local machine or distributed using Kubernetes.

First developed to process ZTF data it is capable to be used for any stream/static data processing pipeline.

Installing apf

apf installation can be done with pip

pip install apf_base

This will install the apf python package and apf command line script.

apf design

apf is based on steps conected through Apache Kafka topics.

Each step is composed by a consumer and is isolated from other steps inside a docker container.

When running, the step calls the execute() method for each message consumed. A step can have multiple producers and databases back-ends plugins that can be accessed inside the execute method to have a more complex logic.

This generic step greatly reduce the development of each component of the pipeline and make it easier to test each component separately.

apf features

  • Automatic Metric Sender (KafkaMetrics)
  • Automatic Code Generation (apf new-step <step_name>)
  • Multiple Consumer Plugins:
    • Kafka
    • AVRO
    • CSV
    • JSON
  • Producers:
    • Kafka
    • CSV
  • Metrics:
    • Kafka

apf Quickstart

A quick-start guide to create a new step can be found here.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

apf_base-1.0.13.tar.gz (15.6 kB view hashes)

Uploaded Source

Built Distribution

apf_base-1.0.13-py3-none-any.whl (23.7 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page