Kafka integration for bluesky.
Free software: 3-clause BSD license
Features
BlueskyConsumer
MongoConsumer
Publisher
RemoteDispatcher
Test
Install docker and docker-compose.
Start a Kafka server:
$ cd bluesky-kafka/scripts $ sudo docker-compose -f bitnami-kafka-docker-compose.yml up
Run tests:
$ cd bluesky-kafka $ pytest
Optionally increase logging output to the console by specifying a logging level:
$ pytest --log-cli-level=DEBUG
Run a Mongo Consumer Group
Create a conda environment:
$ conda create -n consumers python=3.8 $ conda activate consumers
Install packages:
$ pip install bluesky-kafka supervisor
Setup environment variables: mongo_uri reference: https://docs.mongodb.com/manual/reference/connection-string/ bootstrap_servers: comma-separated list of brokers.
$ export BLUESKY_MONGO_URI="mongodb://username:password@machine1:port1,machine2:port2,machine3:port3 $ export KAFKA_BOOTSTRAP_SERVERS="machine1:9092, machine2:9092, machine3:9092"
Update the bluesky_kafka/supervisor/supervisord.conf file with the correct path for your installation.
Start the consumer processes:
$ supervisord -c bluesky_kafka/supervisor/supervisord.conf
Monitor the consumer processes:
$ supervisorctl -c bluesky_kafka/supervisor/supervisorctl.conf
Release files for bluesky-kafka 0.10.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| bluesky-kafka-0.10.0.tar.gz | 45.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| bluesky_kafka-0.10.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 82.0 kB
Release files / bluesky-kafka-0.10.0.tar.gz
| Download URL | bluesky-kafka-0.10.0.tar.gz |
|---|---|
| Size | 45.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5aab55d4d47261327d0ab6c7f76d2b079f93f625917ee118dd9e638b94f1929f
|
|
BLAKE2b-256 checksum How to use checksums |
3f3efd7381a7e712238a0fa0bc8a0feb5c2febd7b9422e4b74951a541affb5e0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.9
|
Release files / bluesky_kafka-0.10.0-py3-none-any.whl
| Download URL | bluesky_kafka-0.10.0-py3-none-any.whl |
|---|---|
| Size | 37.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8b82605d61213d9484e68c20a63d03f24246a775b3729fdc3fcb7259ccac5178
|
|
BLAKE2b-256 checksum How to use checksums |
b1d3a3e8393cdf6e40e94b3a91ab8afdb010769829b520b2fbfdc4697f31a1c3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.9
|
