Skip to main content

Python library for managing and persisting named Python Queues with a variety of backend choices

Project description


A python library for managing and persisting named Python Queues


pip install pydq

Some backends may require extra packages:

pip install pydq[aws]

Basic Usage

with dq_provider('name') as dq:
    # Query the last 10 entries with qid='item_id'
    dq(qid='item_id', start_time=datetime(1, 1, 1), end_time=datetime.utcnow(), limit=10)
    item = dq.get()
    # queue items should be dictionaries with 3 keys: qid, ts, and val
    dq.put({'qid': 'item_id', 'ts': '2019-09-11T03:23:54Z', 'val': 'foo'})

note: specifying qid in the query is optional, if omitted all items between the start and end time are returned

Supported Backends:

Local Storage

  • SQLite:
from dq.local import SQLite as dq_provider

Cloud Storage

  • Buffered DynamoDB
from import BufferedDDB as dq_provider

note: cloud storage is created lazily; the first call to one of these providers will create the resources in the cloud.

Additional API

Some queue methods accept a forget argument that will tell the queue object whether the operation should be persisted. This always defaults to False.

dq.get(forget=True)  # the returned item will be deleted from persistence
dq.put(item, forget=True)    # item will not be persisted
dq.get_all(forget=True)  # Easy way to delete all items in a queue
dq.put_all(items, forget=False)
dq.get_to_dataframe()    # install with pandas extra. Returned items are not deleted
dq.put_from_dataframe(df)    # items will be persisted
dq.clear(forget=True)    # another way to delete all items in a queue

Configuration queues

dq is great for managing simple JSON configuration too. And we get change tracking for free!

from dq.helpers import DqConfig
from dq.local import SQLite
with DqConfig(name='ApplicationConfig', dq_provider=SQLite) as config:
    app_id = config['app_id']
    last_start = config['last_start']
    config['app_status'] = '#1best'

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pydq, version 0.1.18
Filename, size File type Python version Upload date Hashes
Filename, size pydq-0.1.18-py3-none-any.whl (9.0 kB) File type Wheel Python version py3 Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page