High level pub/sub package
Project description
Papfa
Papfa is a high level pub/sub pattern Python library.
Quick Start for Django
Install Papfa:
$ pip install papfa
Add Papfa to your settings:
PAPFA = {
'BROKER': 'KAFKA',
'KAFKA_BOOTSTRAP_SERVERS': ...,
'KAFKA_SASL_PASSWORD': ...,
'KAFKA_SASL_USERNAME': ...,
'KAFKA_SASL_MECHANISM': ...,
'KAFKA_SECURITY_PROTOCOL': ...,
'GROUP_ID_PREFIX': ...,
'SCHEMA_REGISTRY_URL': ...,
'SCHEMA_REGISTRY_BASIC_AUTH': ...,
'CONSUMER_MIDDLEWARES': [
...
]
}
Add consumer in <app>/consumers.py:
from papfa.consumers import consumer
from papfa.dtos import Record
@consumer(topic='topic_name')
def my_consumer(messages: List[Record]):
print(messages)
Produce Message:
from dataclasses import dataclass
from dataclasses_avroschema import AvroModel
from papfa.producer import get_message_producer
from papfa.dtos import Record
@dataclass
class User(AvroModel):
name: str
age: int
r1 = Record(value=User(name='thom', age=53))
r1 = Record(value=User(name='jonny', age=50))
message_producer = get_message_producer(topic='topic_name', User)
message_producer.produce(messages=[r1, r2])
Middleware
Papfa provides middlewares for both consumers and producers. You can implement your own middleware by extending the papfa.middlewares.consumer.ConsumerMiddleware and papfa.middlwares.producer.ProducerMiddleware class.
Default Middlewares
papfa.middlewares.consumer.ConsumedMessageStatsMiddleware - Logs the last message consumed by each topic - consumer group
Serialization
For Now Papfa only support confluent avro serialization with schema registry.
Broker
For Now Papfa only support Apache Kafka.
Features
Batch Processing (Commit per batch)
Consumed Messages Stats
Todos
Handle Idempotency
Add Other Brokers & Serializers
Handle Multiple Broker Cluster
Credits
This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.
History
0.1.0 (2022-05-11)
First release on PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for papfa-0.1.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f5abd404797f9effa64a707afbf440f19a38ae17177d565210e8ebe3f47db66f |
|
MD5 | 1c4157e8f677337f12bdebc1c52da6b9 |
|
BLAKE2b-256 | 3dd4de2b7d5b25a61f9efb1ef3cdc18e9bb4ac4e4aeee5b4fb3d0806c00b5804 |