Base consumer class for working with Avro datums
Project description
avroconsumer
============
A set of `Rejected <https://rejected.readthedocs.io/en/latest/>`_ classes that automatically
encode and decode AMQP message bodies as `Avro <http://avro.apache.org/docs/1.7.7/>`_ datums.
For applications that can share schema files, Avro datum provide small, contract based binary
serialization format. Leveraging AMQP's ``Type`` message property to convey the Avro schema
file for decoding the datum, avroconsumer's classes extend Rejected's
``rejected.consumer.SmartPublishingConsumer``.
|Version| |Downloads| |License| |Status| |Coverage|
Installation
------------
avroconsumer is available on the `Python package index <https://pypi.python.org/pypi/avroconsumer>`_.
Usage
-----
avroconsumer has two base consumer types: ``LocalSchemaConsumer`` and ``RemoteSchemaConsumer``.
The difference between the two resides in how they load the Avro schema file. The
``LocalSchemaConsumer`` loads schema files from a local directory, while the ``RemoteSchemaConsumer``
loads schema files from a remote location accessible via HTTP.
LocalSchemaConsumer
```````````````````
To use the ``LocalSchemaConsumer``, you need to set the ``schema_path`` config value in the consumer
configuration of the rejected configuration file. The following snippet demonstrates an example
configuration:
.. code:: yaml
Consumers:
apns_push:
consumer: app.Consumer
connections: [rabbit1]
qty: 1
queue: datum
qos_prefetch: 1
ack: True
max_errors: 5
config:
schema_path: /etc/avro-schemas/
In this example configuration, if messages are published with a AMQP ``type`` message property of
``foo`` and a ``content-type`` property of ``application/vnd.apache.avro.datum``, classes
extending the combination of ``LocalSchemaConsumer`` will use the Avro schema file located at
``/etc/avro-schemas/foo.avsc`` to deserialize messages.
RemoteSchemaConsumer
````````````````````
The ``RemoteSchemaConsumer`` loads schema files from a remote HTTP server. It expects the
``schema_uri_format`` setting in the consumer configuration of the rejected configuration file.
The following snippet demonstrates an example configuration:
.. code:: yaml
Consumers:
apns_push:
consumer: app.Consumer
connections: [rabbit1]
qty: 1
queue: datum
qos_prefetch: 1
ack: True
max_errors: 5
config:
schema_uri_format: http://schema-server/avro/{0}.avsc
In this example configuration, if messages are published with a AMQP ``type`` message property
of ``foo`` and a ``content-type`` property of ``application/vnd.apache.avro.datum``, classes
extending the combination of ``RemoteSchemaConsumer`` will use the Avro schema file located at
``http://schema-server/avro/foo.avsc`` to deserialize messages.
Example
-------
The following example uses the ``RemoteSchemaConsumer`` class to receive a message and
deserialize it. It evaluates the content of the message and if the field ``foo`` equals
``bar`` it will publish a ``bar`` message.
.. code:: python
import avroconsumer
class FooConsumer(avroconsumer.RemoteSchemaConsumer):
def process(self):
if self.body['foo'] == 'bar':
self.publish('bar', 'amqp.direct', 'routing-key',
{'timestamp': time.time()}, {'bar': True})
Enforcing Message Type
----------------------
As with any instance of ``rejected.consumer.Consumer``, the avroconsumer classes
can automatically rejected messages based upon the ``type`` message property.
Simply set the ``MESSAGE_TYPE`` attribute of your consumer and any messages
received that do not match that message type will be rejected.
Requirements
------------
- `fastavro <https://pypi.python.org/pypi/fastavro>`_
- `rejected <https://pypi.python.org/pypi/rejected>`_
.. |Version| image:: https://img.shields.io/pypi/v/avroconsumer.svg?
:target: https://pypi.python.org/pypi/avroconsumer
.. |Status| image:: https://img.shields.io/travis/gmr/avroconsumer.svg?
:target: https://travis-ci.org/gmr/avroconsumer
.. |Coverage| image:: https://img.shields.io/codecov/c/github/gmr/avroconsumer.svg?
:target: https://codecov.io/github/gmr/avroconsumer?branch=master
.. |Downloads| image:: https://img.shields.io/pypi/dm/avroconsumer.svg?
:target: https://pypi.python.org/pypi/avroconsumer
.. |License| image:: https://img.shields.io/pypi/l/avroconsumer.svg?
:target: https://pypi.python.org/pypi/avroconsumer
============
A set of `Rejected <https://rejected.readthedocs.io/en/latest/>`_ classes that automatically
encode and decode AMQP message bodies as `Avro <http://avro.apache.org/docs/1.7.7/>`_ datums.
For applications that can share schema files, Avro datum provide small, contract based binary
serialization format. Leveraging AMQP's ``Type`` message property to convey the Avro schema
file for decoding the datum, avroconsumer's classes extend Rejected's
``rejected.consumer.SmartPublishingConsumer``.
|Version| |Downloads| |License| |Status| |Coverage|
Installation
------------
avroconsumer is available on the `Python package index <https://pypi.python.org/pypi/avroconsumer>`_.
Usage
-----
avroconsumer has two base consumer types: ``LocalSchemaConsumer`` and ``RemoteSchemaConsumer``.
The difference between the two resides in how they load the Avro schema file. The
``LocalSchemaConsumer`` loads schema files from a local directory, while the ``RemoteSchemaConsumer``
loads schema files from a remote location accessible via HTTP.
LocalSchemaConsumer
```````````````````
To use the ``LocalSchemaConsumer``, you need to set the ``schema_path`` config value in the consumer
configuration of the rejected configuration file. The following snippet demonstrates an example
configuration:
.. code:: yaml
Consumers:
apns_push:
consumer: app.Consumer
connections: [rabbit1]
qty: 1
queue: datum
qos_prefetch: 1
ack: True
max_errors: 5
config:
schema_path: /etc/avro-schemas/
In this example configuration, if messages are published with a AMQP ``type`` message property of
``foo`` and a ``content-type`` property of ``application/vnd.apache.avro.datum``, classes
extending the combination of ``LocalSchemaConsumer`` will use the Avro schema file located at
``/etc/avro-schemas/foo.avsc`` to deserialize messages.
RemoteSchemaConsumer
````````````````````
The ``RemoteSchemaConsumer`` loads schema files from a remote HTTP server. It expects the
``schema_uri_format`` setting in the consumer configuration of the rejected configuration file.
The following snippet demonstrates an example configuration:
.. code:: yaml
Consumers:
apns_push:
consumer: app.Consumer
connections: [rabbit1]
qty: 1
queue: datum
qos_prefetch: 1
ack: True
max_errors: 5
config:
schema_uri_format: http://schema-server/avro/{0}.avsc
In this example configuration, if messages are published with a AMQP ``type`` message property
of ``foo`` and a ``content-type`` property of ``application/vnd.apache.avro.datum``, classes
extending the combination of ``RemoteSchemaConsumer`` will use the Avro schema file located at
``http://schema-server/avro/foo.avsc`` to deserialize messages.
Example
-------
The following example uses the ``RemoteSchemaConsumer`` class to receive a message and
deserialize it. It evaluates the content of the message and if the field ``foo`` equals
``bar`` it will publish a ``bar`` message.
.. code:: python
import avroconsumer
class FooConsumer(avroconsumer.RemoteSchemaConsumer):
def process(self):
if self.body['foo'] == 'bar':
self.publish('bar', 'amqp.direct', 'routing-key',
{'timestamp': time.time()}, {'bar': True})
Enforcing Message Type
----------------------
As with any instance of ``rejected.consumer.Consumer``, the avroconsumer classes
can automatically rejected messages based upon the ``type`` message property.
Simply set the ``MESSAGE_TYPE`` attribute of your consumer and any messages
received that do not match that message type will be rejected.
Requirements
------------
- `fastavro <https://pypi.python.org/pypi/fastavro>`_
- `rejected <https://pypi.python.org/pypi/rejected>`_
.. |Version| image:: https://img.shields.io/pypi/v/avroconsumer.svg?
:target: https://pypi.python.org/pypi/avroconsumer
.. |Status| image:: https://img.shields.io/travis/gmr/avroconsumer.svg?
:target: https://travis-ci.org/gmr/avroconsumer
.. |Coverage| image:: https://img.shields.io/codecov/c/github/gmr/avroconsumer.svg?
:target: https://codecov.io/github/gmr/avroconsumer?branch=master
.. |Downloads| image:: https://img.shields.io/pypi/dm/avroconsumer.svg?
:target: https://pypi.python.org/pypi/avroconsumer
.. |License| image:: https://img.shields.io/pypi/l/avroconsumer.svg?
:target: https://pypi.python.org/pypi/avroconsumer
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
avroconsumer-1.1.0.tar.gz
(5.8 kB
view hashes)
Built Distribution
Close
Hashes for avroconsumer-1.1.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2a50bd4927474813e7faea5b0d93913bcedc487875fd38123eeb9db593f81912 |
|
MD5 | dff1d2a129544a2d2bdf692060548676 |
|
BLAKE2b-256 | e1c1f2bbcfe72e0a403896f45aeb26e511e1a677613f83ce937c425cff15ddd1 |