Skip to main content

Infrastructure components for automated data processing at Diamond Light Source

Project description

Zocalo

PyPI release Conda version Build status Language grade: Python Total alerts Documentation status Supported Python versions Code style: black BSD license

Zocalo is an automated data processing system designed at Diamond Light Source. This repository contains infrastructure components for Zocalo.

The idea of Zocalo is a simple one - to build a messaging framework, where text-based messages are sent between parts of the system to coordinate data analysis. In the wider scope of things this also covers things like archiving, but generally it is handling everything that happens after data aquisition.

Zocalo as a wider whole is made up of two repositories (plus some private internal repositories when deployed at Diamond):

  • DiamondLightSource/python-zocalo - Infrastructure components for automated data processing, developed by Diamond Light Source. The package is available through PyPi and conda-forge.
  • DiamondLightSource/python-workflows - Zocalo is built on the workflows package. It shouldn’t be necessary to interact too much with this package, as the details are abstracted by Zocalo. workflows controls the logic of how services connect to each other and what a service is, and actually send the messages to a message broker. Currently this is an ActiveMQ broker (via STOMP) but support for a RabbitMQ broker (via pika) is being added. This is also available on PyPi and conda-forge.

As mentioned, Zocalo is currently built on top of ActiveMQ. ActiveMQ is an apache project that provides a message broker server, acting as a central dispatch that allows various services to communicate. Messages are plaintext, but from the Zocalo point of view it’s passing aroung python objects (json dictionaries). Every message sent has a destination to help the message broker route. Messages may either be sent to a specific queue or broadcast to multiple queues. These queues are subscribed to by the services that run in Zocalo. In developing with Zocalo, you may have to interact with ActiveMQ or RabbitMQ, but it is unlikely that you will have to configure it.

Zocalo allows for the monitoring of jobs executing python-workflows services or recipe wrappers. The python-workflows package contains most of the infrastructure required for the jobs themselves and more detailed documentation of its components can be found in the python-workflows GitHub repository and the Zocalo documentation.

History

Unreleased

0.14.0 (2021-12-14)

  • zocalo.dlq_purge offers a --location flag to override where files are being written
  • zocalo.dlq_reinject can again understand zocalo.dlq_purge output passed on stdin
  • Reinjected messages now carry a dlq-reinjected: True header field

0.13.0 (2021-12-01)

  • zocalo.queue_drain now allows the automatic determination of destination queues for recipe messages
  • zocalo.queue_drain fixed for use in a RabbitMQ environment
  • zocalo.dlq_purge fixed for use in a RabbitMQ environment
  • New functions in zocalo.util to easily annotate log messages with system context information

0.12.0 (2021-11-15)

  • Add support for queue/exchange bindings to RabbitMQAPI
  • Drop support for Python 3.6 and 3.7

0.11.1 (2021-11-08)

  • Add a RabbitMQ HTTP API in zocalo.util.rabbitmq

0.11.0 (2021-11-03)

  • Add command line tools for handling dead-letter messages
  • zocalo.dlq_check checks dead-letter queues for messages
  • zocalo.dlq_purge removes messages from specified DLQs and dumps them to a directory specified in the Zocalo configuration
  • zocalo.dlq_reinject takes a serialised message produced by zocalo.dlq_purge and places it back on a queue
  • Use argparse for all command line tools and make use of workflows transport argument injection. Minimum workflows version is now 2.14
  • New zocalo.util.rabbitmq.RabbitMQAPI() providing a thin wrapper around the RabbitMQ HTTP API

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for zocalo, version 0.14.0
Filename, size File type Python version Upload date Hashes
Filename, size zocalo-0.14.0.tar.gz (159.5 kB) File type Source Python version None Upload date Hashes View
Filename, size zocalo-0.14.0-py3-none-any.whl (45.2 kB) File type Wheel Python version py3 Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page