Python library for managing and persisting named Python Queues with a variety of backend choices
Project description
dq
A python library for managing and persisting named Python Queues
Installation
pip install pydq
Some backends may require extra packages:
pip install pydq[aws]
Basic Usage
with dq_provider('name') as dq:
# Query the last 10 entries with qid='item_id'
dq(qid='item_id', start_time=datetime(1, 1, 1), end_time=datetime.utcnow(), limit=10)
item = dq.get()
# queue items should be dictionaries with 3 keys: qid, ts, and val
dq.put({'qid': 'item_id', 'ts': '2019-09-11T03:23:54Z', 'val': 'foo'})
note: specifying qid
in the query is optional, if omitted all items between the start and end time are returned
Supported Backends:
Local Storage
- SQLite:
from dq.local import SQLite as dq_provider
Cloud Storage
- Buffered DynamoDB
from dq.cloud import BufferedDDB as dq_provider
note: cloud storage is created lazily; the first call to one of these providers will create the resources in the cloud.
Additional API
Some queue methods accept a forget
argument that will tell the queue object whether the operation should be persisted. This always defaults to False
.
dq.get(forget=True) # the returned item will be deleted from persistence
dq.put(item, forget=True) # item will not be persisted
dq.get_all(forget=True) # Easy way to delete all items in a queue
dq.put_all(items, forget=False)
dq.get_to_dataframe() # install with pandas extra. Returned items are not deleted
dq.put_from_dataframe(df) # items will be persisted
dq.clear(forget=True) # another way to delete all items in a queue
Configuration queues
dq is great for managing simple JSON configuration too. And we get change tracking for free!
from dq.helpers import DqConfig
from dq.local import SQLite
with DqConfig(name='ApplicationConfig', dq_provider=SQLite) as config:
app_id = config['app_id']
last_start = config['last_start']
config['app_status'] = '#1best'
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.