A simple library to process a list of messages from disk
Project description
Disk Queue Processing
Library to do simple disk based processing of messagepack dictionaries in a file.
All files are flat files and directories. To manage a simple folder structure with naming convention, use the Project
class.
From the project you can open a source/sink and read/write with them using python dictionaries.
Sinks are rotated, sources keep track of a last read entry to allow you to continue later. To do this on close, use the Project class.
from dqp.queue import Project
with Project("/tmp/banana") as project:
s = project.open_sink("hello")
s.write_dict({"a": 1})
s.write_dict({"b": 1})
s.write_dict({"c": 1})
s.write_dict({"d": 1})
with Project("/tmp/banana") as project:
s = project.continue_source("hello")
for filename, index, msg in s:
print("1st go:", msg)
break
with Project("/tmp/banana") as project:
s = project.continue_source("hello")
for filename, index, msg in s:
print("2nd go:", msg)
Queue files are rotated based on timestamp, so each write_dict does look at the clock to see if we already need to rotate and what the new file path should be.
Clean up by telling the source to unlink everything up to last or a given queue filename prefix.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.