Real-time application in order to dominate Humans.
Project description
Domination
Real-time application made to monitor and dominate Humans.
From the rating of every human (from 1 to 100) sent to the kafka topic dominate
,
we detect, in real time, which type they are:
Sha
if its rating is a multiple of 3Dow
if its rating is a multiple of 5ShaDow
if its rating is a multiple of 3 and 5Incompatible
if none of the above.
Using a new kafka topic shadow
, we make the result available to a clickhouse
table named shadow
.
System Design
+----------------+ +-------------+ +------------------+
| domination | | dominate | | domination |
+----------------+ +-------------+ +------------------+
| python | +-----------> | | +--------> | python |
| Faust producer | | Kafka topic | | Faust agent |
| HumanRatings | | | | HumanCategorized |
+----------------+ +-------------+ +------------------+
+
+---------------------------------------------------------------+
v
+----------------+ +-------------------+ +-------------------+
| shadow | | shadow_stream | | shadow_consumer |
+----------------+ +-------------------+ +-------------------+
| | +------> | clickhouse table | +------> | clickhouse table |
| Kafka topic | | encapsulate topic | | materialized view |
+----------------+ +-------------------+ +-------------------+
+
+---------------------------------------------------------------+
v
+-------------------+
| shadow |
+-------------------+
| clickhouse table |
| store rows |
+-------------------+
Structure of Kafka messages:
-
topic
dominate
:{"rating": <integer>, "unique_id": "<string>"}
-
topic
shadow
:{"type": <integer>, "unique_id": "<string>", "emit_timestamp": <datetime>}
Usage
Requirements
- Python >= 3.6
- docker-compose
Start domination
git clone https://github.com/AzemaBaptiste/domination.git && cd domination
virtualenv -p python3.8 venv && source venv/bin/activate
pip install domination
docker-compose up -d
./create_clickhouse_tables.sh
domination worker -l info
Explore data in a new terminal
docker exec -it clickhouse bin/bash -c "clickhouse-client --multiline"
> SELECT COUNT(type) AS COUNT, type FROM shadow
GROUP BY type ORDER BY (COUNT) DESC LIMIT 10;
Stop domination
Ctrl + C (stop faust app)
docker-compose down
Run The Algorithm
You can also run The Algorithm as a standalone. It will print the type of every human rated from 1 to 1337.
python the_algorithm.py
Development
# Install
virtualenv -p python3.8 venv
source venv/bin/activate
pip install -r requirements.txt
make install
# Build
make test # coverage tests
make linter # runs pylint
make build
Explore data
Open CLI of the clickhouse client
docker exec -it clickhouse bin/bash -c "clickhouse-client --multiline"
Description of clickhouse tables
shadow_stream
: consumer of the kafka topic
CREATE TABLE IF NOT EXISTS shadow_stream
(
`type` String,
`unique_id` String,
`emit_timestamp` DateTime
) ENGINE = Kafka()
SETTINGS
kafka_broker_list = 'kafka:29092',
kafka_topic_list = 'shadow',
kafka_group_name = 'shadow-group',
kafka_format = 'JSONEachRow',
kafka_skip_broken_messages = 1;
shadow
: final table
CREATE TABLE shadow as shadow_stream
ENGINE = MergeTree()
PARTITION BY toYYYYMM(emit_timestamp)
ORDER BY type;
shadow_consumer
: materialized view feeding the final table with the consumer table
CREATE MATERIALIZED VIEW shadow_consumer
TO shadow
AS SELECT * FROM shadow_stream;
References
TODO
- deploy package to pypi using github actions
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.