ampel-core 0.8.3b15

Alice in Modular Provenance-Enabled Land

AMPEL-core

Introduction

AMPEL is a modular and scalable platform with explicit provenance tracking, suited for systematically processing large - possibly complex and heterogeneous - datasets in real-time or not. This includes selecting, analyzing, updating, combining, enriching and reacting to data.

The framework requires analysis and reaction logic to be broken down in adequate indepentent units. AMPEL is general enough to be applicable in various fields, it was originaly developped to solve challenges in the context of experimental astrophysics.

AMPEL is written in Python 3.10 and its codebase is fully typed.

Architecture

Tiers

AMPEL is made of four execution layers (tiers) that replace a traditional pipeline architecture.

The tiers are independently scheduled and the information exchange between tiers occurs via a dedicated database. The execution layer architecture along with the database structure allows for simple parallelization.

Units

Each tier is modular and executes so-called "units".

Ampel base units have standardized inputs and ouputs, enforced through abstract classes which units inherit.

Processes

Every change in AMPEL is triggered by a process. A process executes, at a given date and time, a processor unit that itself runs one or multiple base units with specific configurations. Information about process executions are registred into the database. The majority of processes are associated with a specific tier but general processes are possible.

A working AMPEL system will spawn multiple processes, posssibly concurently, accross the four AMPEL tiers. This will result in the ingestion and analysis of data and the triggering of automated reactions when given data states are detected.

Channels

Channels are convenient for multi-user or multi-prupose AMPEL systems. They allow to define and enforce access rights and to privatize processes, meaning that the output generated by the processes will be only accessible to processes belonging to the same channel.

Internally, channels are just tags in database documents and ampel configuration files.
From a user perspective, a channel can be seen as a collection of private processes.

Processes associated with a given channel

Note that within AMPEL, different channels requiring the same computation will not result in the required computation being performed twice.

Repositories

The AMPEL code is partitioned in different repositories.
The only mandatory repository in this list is ampel-interface

Public abstract class definitions:
https://github.com/AmpelProject/Ampel-interface

Specialized classes for Tier 0, capable of handling alerts:
https://github.com/AmpelProject/Ampel-alerts

An add-on that introduces two generic classes of datapoints:
https://github.com/AmpelProject/Ampel-photometry

Example of an instrument specific implementation:
https://github.com/AmpelProject/Ampel-ztf

Numerous base units, the majority being specific to astronomy:
https://github.com/AmpelProject/Ampel-contrib-HU/

Database

MongoDB is used to store data. The collections have been designed and indexed for fast insertion and query. Users do not interact with the database directly. Information exchange is instead regulated through (python) abstract base classes from which units are constructed. A specific set of internal classes handle database input and output.

Containers

All AMPEL software, can be combined into one container that defines an instance. These containers can be used both to process real-time data as well as to reprocess archived data. The containers themselves should be archived as well.

Installing Ampel

The latest release of ampel-core can be installed from PyPI with pip, e.g.:

pip install ampel-core

Other projects like ampel-alerts, ampel-ztf, etc. can also be installed with pip.

Development

ampel-core uses poetry for dependency management and packaging. To work with an editable install it's recommended that you set up poetry and install ampel-core in a virtual environment by doing

poetry install

Alternatively, may also use a setuptools-style editable install from setup.py:

pip install -e .

Note that setup.py and requirements.txt are auto-generated; any changes you commit will be overwritten the next time pyproject.toml is updated.