A package to allow flexible and quick programs on a message bus/kafka.
Proof of concept and example for smart routing on a dumb bus.
This is a small project focused on my blog posts around routing on a message bus that's dumb. Like Kafka.
I use some of the cacheing ideas in here for a few of my data lookups.
Once you download the clone/download this repo, let's setup some stuff in the test path:
- git clone email@example.com:chrisfauerbach/metrobus.git
- cd metrobus
- cd test
- pip install --upgrade pipenv
- pipenv shell
- mkdir data
- python make_whitelist.py > data/whitelist.dat
- python make_account_email.py > data/emails.dat
You'll then see a lot of docker compose stuff happening. The environment variables are in metrobus.env
They're set to:
- Kafka: kafka:9092
- Redis: redis:6379
Those host names are set in the docker-compose file.
Web application, in order to see some super basic stats from redis.
Flask app in web.py
Cool name, eh? Pusher?
This is the example 'source' of data
Raw JSON messages going into Kafka...
This is the first major step of the process.
Bus driver is my smart router dude. In a real
application, bus driver could/would inspect
the state of the data coming in. THen, it would
dynamically build a route
#Contact Point - Add Email (Stage one)
This application uses the fake account ID
generated in the pusher. Adds an email
address based on the cps.dat file you
previously generated. Drops message if can't find
Whitelist (stage two)
Example of a whitelist. I blogged about black list.
Same dealio? If record (email in this case) isn't
in the list, then we can drop the messsage (return)
Log Stop (stage three)
This is an example 'sink' stop on the metrobus
All this bad boy does is logs the message..
in the real world, this could shove into elasticsearch
or something less cool than ES.
This is a personal project and has nothing to do with my employer or any contracts/clients I have.
I will build this project out to be MIT licensed though, for anyone to use and abuse.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size & hash SHA256 hash help||File type||Python version||Upload date|
|metrobus-0.0.6-py3-none-any.whl (5.6 kB) Copy SHA256 hash SHA256||Wheel||py3||Jul 13, 2018|
|metrobus-0.0.6.tar.gz (5.4 kB) Copy SHA256 hash SHA256||Source||None||Jul 13, 2018|