Skip to main content

Kafka integration for bluesky.

Project description

https://github.com/bluesky/bluesky-kafka/actions/workflows/tests.yml/badge.svg https://img.shields.io/pypi/v/bluesky-kafka.svg

Kafka integration for bluesky.

  • Free software: 3-clause BSD license

Features

  • BlueskyConsumer
  • MongoConsumer
  • Publisher
  • RemoteDispatcher

Release History

v0.3.0 (2020-09-03)

  • added BlueskyConsumer
  • added MongoConsumer
  • added supervisor configuration file for mongo_normalized_consumer.py
  • rewrote RemoteDispatcher to use BlueskyConsumer
  • changed default serialization method to MessagePack

v0.4.0 (2021-04-09)

  • added continue_polling parameter to BlueskyConsumer.start()
  • added utils.py
  • added BlueskyKafkaException
  • split tests into multiple files
  • create and clean up topics for each test
  • simplified produce/consume tests to run in one process
  • configured live logging in pytest.ini
  • switched from travis to github for continuous integration

Test

Install docker and docker-compose.

Start a Kafka server:

$ cd bluesky-kafka/scripts
$ sudo docker-compose -f bitnami-kafka-docker-compose.yml up

Run tests:

$ cd bluesky-kafka
$ pytest

Optionally increase logging output to the console by specifying a logging level:

$ pytest --log-cli-level=DEBUG

Run a Mongo Consumer Group

Create a conda environment:

$ conda create -n consumers python=3.8
$ conda activate consumers

Install packages:

$ pip install bluesky-kafka supervisor

Setup environment variables: mongo_uri reference: https://docs.mongodb.com/manual/reference/connection-string/ bootstrap_servers: comma-separated list of brokers.

$ export BLUESKY_MONGO_URI="mongodb://username:password@machine1:port1,machine2:port2,machine3:port3
$ export KAFKA_BOOTSTRAP_SERVERS="machine1:9092, machine2:9092, machine3:9092"

Update the bluesky_kafka/supervisor/supervisord.conf file with the correct path for your installation.

Start the consumer processes:

$ supervisord -c bluesky_kafka/supervisor/supervisord.conf

Monitor the consumer processes:

$ supervisorctl -c bluesky_kafka/supervisor/supervisorctl.conf

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for bluesky-kafka, version 0.4.0
Filename, size File type Python version Upload date Hashes
Filename, size bluesky_kafka-0.4.0-py3-none-any.whl (21.3 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size bluesky-kafka-0.4.0.tar.gz (35.1 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page