Skip to main content

Thread-safe, mixed sync-async queue for Python

Project description

newt

Thread-safe, mixed-threading-and-asyncio, producer-consumer queue for Python.

Heavily based on janus, but newt lazily initializes event loop which makes the queue much more flexible:

  • newt.Queue could be initialized outside subthread or coroutine
  • supports information exchange between a thread and a coroutine
  • ensures thread-safety

Install

$ pip install newt

Usage

Suppose there is a threaded target function which produces items, and a coroutine which consumes items.

from newt import Queue


def threaded(sync_queue):
    for i in range(100):
        sync_queue.put(i)
    sync_queue.join()

sync_queue follows the interface of Python built-in synchronized queue class

async def coroutine(async_queue):
    for i in range(100):
        assert await async_queue.get() == i
        async_queue.task_done()

async_queue follows the vanilla Python asyncio.Queue

Thread in an executor -> Coroutine

The following example shows how to produce item in a thread which executed in the executor, and consume the item in a coroutine.

import asyncio

loop = asyncio.get_event_loop()


async def main():
    future = loop.run_in_executor(None, threaded, queue.sync_queue)
    await coroutine(queue.async_queue)
    await future

    queue.close()
    await queue.wait_closed()

loop.run_until_complete(main())

Normal thread -> Coroutine

newt.Queue also supports to produce item in a normal threading,

loop = asyncio.get_event_loop()


async def main():
    await coroutine(queue.async_queue)
    queue.close()
    await queue.wait_closed()

t = threading.Thread(target=threaded, args=(queue.sync_queue,))
t.start()

loop.run_until_complete(main())

License

MIT

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

newt-0.1.0.tar.gz (9.3 kB view hashes)

Uploaded Source

Built Distribution

newt-0.1.0-py3-none-any.whl (9.5 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page