Skip to main content

Thread-safe, mixed sync-async queue for Python

Project description

newt

Thread-safe, mixed-threading-and-asyncio, producer-consumer queue for Python.

Heavily based on janus, but newt lazily initializes event loop which makes the queue much more flexible:

  • newt.Queue could be initialized outside subthread or coroutine
  • supports information exchange between a thread and a coroutine
  • ensures thread-safety

Install

$ pip install newt

Usage

Suppose there is a threaded target function which produces items, and a coroutine which consumes items.

from newt import Queue


def threaded(sync_queue):
    for i in range(100):
        sync_queue.put(i)
    sync_queue.join()

sync_queue follows the interface of Python built-in synchronized queue class

async def coroutine(async_queue):
    for i in range(100):
        assert await async_queue.get() == i
        async_queue.task_done()

async_queue follows the vanilla Python asyncio.Queue

Thread in an executor -> Coroutine

The following example shows how to produce item in a thread which executed in the executor, and consume the item in a coroutine.

import asyncio

loop = asyncio.get_event_loop()


async def main():
    future = loop.run_in_executor(None, threaded, queue.sync_queue)
    await coroutine(queue.async_queue)
    await future

    queue.close()
    await queue.wait_closed()

loop.run_until_complete(main())

Normal thread -> Coroutine

newt.Queue also supports to produce item in a normal threading,

loop = asyncio.get_event_loop()


async def main():
    await coroutine(queue.async_queue)
    queue.close()
    await queue.wait_closed()

t = threading.Thread(target=threaded, args=(queue.sync_queue,))
t.start()

loop.run_until_complete(main())

License

MIT

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

newt-0.1.0.tar.gz (9.3 kB view details)

Uploaded Source

Built Distribution

newt-0.1.0-py3-none-any.whl (9.5 kB view details)

Uploaded Python 3

File details

Details for the file newt-0.1.0.tar.gz.

File metadata

  • Download URL: newt-0.1.0.tar.gz
  • Upload date:
  • Size: 9.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.3.1 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.3

File hashes

Hashes for newt-0.1.0.tar.gz
Algorithm Hash digest
SHA256 ee184e66764b9ec3f6fae548f30d5c73a33a7b213a7058523623967d1ce55c64
MD5 12926af78d3ce6a7c3525745a7176119
BLAKE2b-256 82cf64f2db5fa2c218ddf3614d5298a5a2a2c7a58091ccd1f4c29f0fcd6888db

See more details on using hashes here.

File details

Details for the file newt-0.1.0-py3-none-any.whl.

File metadata

  • Download URL: newt-0.1.0-py3-none-any.whl
  • Upload date:
  • Size: 9.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.24.0 setuptools/47.3.1 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.3

File hashes

Hashes for newt-0.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 76910335ef3daef346e694e428e88f26bb64f0f85c3907204c02f3121b088ec7
MD5 78ca14746550633ebf67427a636e9ad4
BLAKE2b-256 2aa0d3b2c47ad22713d5772cc45ee184517a2adee78794e00524b47485d64c85

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page