Safe consume and process data.
Project description
aiosafeconsumer is a library that provides abstractions and some implementations to consume data somewhere and process it.
Abstractions:
DataSource - waits for data and returns batch of records using Python generator
DataProcessor - accepts batch of records and precess it
DataTransformer - accepts batch of records and transform it and calls another processor to precess it. Extends DataProcessor
Worker - abstract worker
ConsumerWorker - connects DataSource and DataProcessor
DataWriter - base abstraction to perform data synchronization
Current implementations:
KafkaSource - read data from Kafka
RedisWriter - synchronize data in Redis
WorkerPool - controller to setup and run workers in parallel. Can handle worker failures and restarts workers when it fails or exits.
Recommend producer library: https://github.com/lostclus/django-kafka-streamer
Example application: https://github.com/lostclus/WeatherApp
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file aiosafeconsumer-0.0.1.tar.gz.
File metadata
- Download URL: aiosafeconsumer-0.0.1.tar.gz
- Upload date:
- Size: 18.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.12.6
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 8380d26850ffd84f0e8e6a91687cc34113698e1b50ae510ec32ed5239f887cd7 |
|
| MD5 | cdc9bae959f13e5a95eb10f1d222037b |
|
| BLAKE2b-256 | 12c8a3004c967c6bf89e9cbfb76ba2ab5e3b6d233886949ae4865d5659701dd3 |
File details
Details for the file aiosafeconsumer-0.0.1-py3-none-any.whl.
File metadata
- Download URL: aiosafeconsumer-0.0.1-py3-none-any.whl
- Upload date:
- Size: 16.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.1 CPython/3.12.6
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | ad52e4e054d31e3f21d13d00d6ae6c49bb71d0b6895b419e382581a7a5ebcaf6 |
|
| MD5 | 5633648a5a9ab3a49e3c6e0a94154765 |
|
| BLAKE2b-256 | 5fc5a145efd4c7870bef0209e7d99b48bdcc474746a42b8255cf9bf4a24f903a |
