This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

VSQS is a message queuing system that is inspired on Amazon’s SQS API, uses the file system as persistent storage and does not require active broker daemons.

Message delivery is strictly FIFO. Consumed messages must explicitly be deleted to prevent automatic requeuing.

File State Transitions

Each queue is represented by a directory on the file system. All messages on a queue are individual files within this directory. There is no queue nesting.

Messages are stored as individual files that transition through the following state diagram from creation to deletion.

File names are the millisecond unix timestamp of creation, followed by an optional extension to indicate the file’s current state. Transitions between states are implemented as atomic file system operations so that multiple processes can safely use a shared queue.

New

When a new message is published a new file is created whose name is the current unix timestamp with the extension .new (e.g. 1415776871123.new).

Files are created atomically: open(fn, O_CREAT|O_EXCL|O_WRONLY)

If creation fails because a concurrent process just wrote a message at the exact same time, the publisher enters a retry loop, updating the file name with the current time, until it succeeds.

If the call is successful, the client then performs a directory listing to check for the presence of files with the same name, but with an extension. If one is found, that indicates the existence of a message that was published at the same time, but is currently in a different state. To avoid clashes, the client then deleted it temporary files and loops back.

Only once a new file is successfully created and there are no other files with the same timestamp in their filename can the message’s payload be written to the new file, after which the file is closed and moved into its final destination by dropping the .new extension.

Ready

Files without extension are ready for consumption. Queues are FIFO and so to consume a message a client performs a directory listing, orders the file names numerically (“99” comes before “100”), ignores all files that have extensions and selects the oldest message.

To consume the message, the client then renames the file by adding the expiration time as extension to the file (e.g. 1415776871123.1415776879654).

The expiration file is calculated by adding the visibility timeout to the current time and expressing it as another unix timestamp with millisecond precision.

The message is now marked as being in the processes of getting consumed. No other consumer will touch the message.

If renaming failed it indicates a race with another client. Just loop back in and select the next message.

Messages are returned to the user application accompanied by their unique message id. This id can be their filename (minus any extensions).

Deleted

When a consumer is done processing a message they must actively delete it by specifying its id.

To delete a message, vsqs then performs a directory listing, looking for files with that name, regardless of extension. This should match zero or one file which is then deleted.

Requeuing

Vsqs uses visibility timeouts similar to SQS. Visibility timeouts essentially hide messages while they’re being processed. If a client dies during processing, the message will be placed back in the queue after its visibility timeout expires.

Requeuing is a synchronous process performed by vsqs consumers while waiting for new messages to consume.

After performing a directory listing, but before dequeuing the next message from the queue, a consumer looks for message files that have a visibility timeout extension that lies in the past (indicating expiration). For each one found, the client then requeues it by simple dropping the extension from the file name (implemented as an atomic rename operation). ENOENT failures indicate a race and can safely be ignored.

After requeuing vsqs loops back, does a new directory listing and starts over. Only when there are no more expired messages can the next message be consumed.

Release History

Release History

0.3

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
vsqs-0.3.tar.gz (6.0 kB) Copy SHA256 Checksum SHA256 Source Nov 30, 2014

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting