Skip to main content
Join the official 2020 Python Developers SurveyStart the survey!

ALeRCE Alert Processing Framework.

Project description

Alert Processing Framework (apf)

Documentation Status codecov

apf is a framework developed to create a dockerized pipeline to process an alert stream, that can be easily be deployed in a local machine or distributed using Kubernetes.

First developed to process ZTF data it is capable to be used for any stream/static data processing pipeline.

Installing apf

apf installation can be done with pip

pip install apf_base

This will install the apf python package and apf command line script.

apf design

apf is based on steps conected through Apache Kafka topics.

Each step is composed by a consumer and is isolated from other steps inside a docker container.

When running, the step calls the execute() method for each message consumed. A step can have multiple producers and databases back-ends plugins that can be accessed inside the execute method to have a more complex logic.

This generic step greatly reduce the development of each component of the pipeline and make it easier to test each component separately.

apf features

  • Automatic Metric Sender (KafkaMetrics)
  • Automatic Code Generation (apf new-step <step_name>)
  • Multiple Consumer Plugins:
    • Kafka
    • AVRO
    • CSV
    • JSON
  • Producers:
    • Kafka
    • CSV
  • Metrics:
    • Kafka

apf Quickstart

A quick-start guide to create a new step can be found here.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for apf-base, version 1.0.1
Filename, size File type Python version Upload date Hashes
Filename, size apf_base-1.0.1.linux-x86_64.tar.gz (25.1 kB) File type Source Python version None Upload date Hashes View
Filename, size apf_base-1.0.1-py3-none-any.whl (22.2 kB) File type Wheel Python version py3 Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page