Skip to main content

A simple message-oriented middleware library built for Python IPC across machine boundaries

Project description

PyMQ

PyPI Version PyPI License Build Status Coverage Status

PyMQ is a simple message-oriented middleware library for implementing Python IPC across machine boundaries. The API enables different styles of remoting via Pub/Sub, Queues, and synchronous RPC.

With PyMQ, developers can integrate Python applications running on different machines in a loosely coupled way over existing transport mechanisms. PyMQ currently provides a Redis backend, a POSIX IPC backend for single-machine IPC, and an in-memory backend for testing. The API is extensible and other transports can be plugged in.

Using PyMQ

Initialize PyMQ

The core module manages a global eventbus instance that provides the remoting primitives. The default Redis implementation uses an event loop over a pubsub object. The global eventbus is initialized via pymq.init and by passing a provider factory.

import pymq
from pymq.provider.redis import RedisConfig

# starts a new thread with a Redis event loop
pymq.init(RedisConfig())

# main application control loop

pymq.shutdown()

This will create an eventbus instance on top of a local Redis server.

Pub/Sub

Pub/Sub allows asynchronous event-based communication. Event classes are used to transport state and identify channels.

import pymq

# common code
class MyEvent:
    pass

# subscribe code
@pymq.subscriber
def on_event(event: MyEvent):
    print('event received')

# publisher code
pymq.publish(MyEvent())

Queues

Queues are straight forward, as they are compatible to the Python queue.Queue specification.

import pymq

queue = pymq.queue('my_queue') 
queue.put('obj')
print(queue.get()) # outputs 'obj'

RPC

Server code

import pymq

@pymq.remote('product_remote')
def product(a: int, b: int) -> int: # pymq relies on type hints for marshalling
    return a * b

Client code

import pymq

product = pymq.stub('product_remote')
product(2, 4) # 8

With a shared code-base, methods can also be exposed and called by passing the callable. For example,

import pymq

# common code
class Remote:
    def echo(self, param) -> None:
        return 'echo: ' + param

# server
obj = Remote()
pymq.expose(obj.echo)

# client
echo = pymq.stub(Remote.echo)
echo('pymq') # "echo: pymq"

If there are multiple providers of the same object, then a stub can be initialized with multi=True to get a list of results. It may be useful to use a timeout in this case.

remote = pymq.stub('remote_method', multi=True, timeout=2)

result = remote() # result will be a list containing the results of all invocations of available remote objects

Providers

  • SimpleEventBus used for testing and rudimentary single-thread dispatching
  • RedisEventBus works across network and process boundaries but requires a running redis instance
  • IpcEventBus uses posix_ipc message queues as event loops and maintains a tree of topic subscriptions in /run/shm. Useful for eventing across process boundaries without an additional server component.

Compatibility

Python 3.6+

Known Limitations

  • JSON serialization relies heavily on type hints. Sending complex types without type hints will cause type errors.
  • There is currently no support for polymorphism with JSON serialization
  • Pattern-based topic matching does not work for the in-memory eventbus or the IPC event bus
  • You can only have a limited number of Queues when using the IPC provider, as the kernel limits the number of file descriptors per process
  • Subscriptions by foreign components to RPC channels will cause issues in multi-call scenarios
  • Using the pymq singleton in multiprocessing scenarios may not work as expected because the module holds a Thread in a global variable. A workaround is to re-start the bus by calling shutdown() and init() in the forked Process.

Background

Originally part of the Symmetry project, was extracted as a standalone library.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pymq, version 0.4.0
Filename, size File type Python version Upload date Hashes
Filename, size pymq-0.4.0-py3-none-any.whl (28.3 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size pymq-0.4.0.tar.gz (22.9 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page