Skip to main content

Python library for managing and persisting named Python Queues with a variety of backend choices

Project description


A python library for managing and persisting named Python Queues


pip install pydq

Some backends may require extra packages:

pip install pydq[aws]

Basic Usage

with dq_provider('name') as dq:
    # Query the last 10 entries with qid='item_id'
    dq(qid='item_id', start_time=datetime(1, 1, 1), end_time=datetime.utcnow(), limit=10)
    item = dq.get()
    # queue items should be dictionaries with 3 keys: qid, ts, and val
    dq.put({'qid': 'item_id', 'ts': '2019-09-11T03:23:54Z', 'val': 'foo'})

note: specifying qid in the query is optional, if omitted all items between the start and end time are returned

Supported Backends:

Local Storage

  • SQLite:
from pydq.local import SQLite as dq_provider

Cloud Storage

  • DynamoDB
from import DynamoDB as dq_provider

note: cloud storage is created lazily; the first call to one of these providers will create the resources in the cloud.

Additional API

Some queue methods accept a forget argument that will tell the queue object whether the operation should be persisted. This always defaults to False.

dq.get(forget=True)  # the returned item will be deleted from persistence
dq.put(item, forget=True)    # item will not be persisted
dq.get_all(forget=True)  # Easy way to delete all items in a queue
dq.put_all(items, forget=False)
dq.get_to_dataframe()    # install with pandas extra. Returned items are not deleted
dq.put_from_dataframe(df)    # items will be persisted
dq.clear()    # another way to delete all items in a queue

Configuration queues

dq is great for managing simple JSON configuration too. And we get change tracking for free!

from pydq.helpers import DqConfig
from pydq.local import SQLite
with DqConfig(name='ApplicationConfig', dq_provider=SQLite) as config:
    app_id = config['app_id']
    last_start = config['last_start']
    config['app_status'] = '#1best'

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Built Distribution

pydq-0.1.20-py3-none-any.whl (11.7 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page