This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

fsq is a standard for implementing queueing structures in POSIX file-systems. fsq provides a standard for both directory layouts and work-item naming, which allow for idempotent work-item construction, atomic enqueueing, and atomic completion.

The fsq python library (import fsq) provides a programattic way to enqueue, scan, introspect and manipulate queues from Python.

The fsq program provides mechanisms for enqueueing, scanning, introspecting, auditing and repairing queues from the command line.

For more on the fsq standard see man 7 fsq, for more on the fsq program see man 1 fsq.

Installing

The fsq project lives on github, and is available via pip.

Installing v0.2.5 From Pip

sudo pip install fsq==0.2.5

Installing v0.2.5 From Source

curl https://github.com/axialmarket/fsq/archive/version_0.2.5.tar.gz | tar vzxf -
cd fsq
sudo python setup.py install

Quick Overview

Installing Queues

To install a queue, simply run:

$ fsq install a_queue

Or via the Python API:

>>> import fsq
>>> fsq.install(a_queue)

Installing a queue will create a directory in FSQ_ROOT (/var/fsq/):

/var/fsq/a_queue
├── done
├── fail
├── queue
└── tmp

Enqueueing Work

To enqueue work to the a_queue queue, simply run:

$ echo "data" | fsq enqueue a_queue args to enqueue

Or from the Python API:

>>> import fsq
>>> # enqueue a string
>>> fsq.senqueue('a_queue', 'data', 'args', 'to', 'enqueue')
>>> # ... or a file
>>> fsq.senqueue('a_queue', '/path/to/data.file', 'args', 'to', 'enqueue')

Enqueueing adds a file to the queue directory of a_queue:

/var/fsq/a_queue
├── done
├── fail
├── queue
|   └── _20131005205643_0_25577_mss_0_args_to_enqueue
└── tmp

Processing Work

To process jobs, use the fsq scan program:

$ # echo gets "args", "to", "enqueue" as $1..$3 and "data" on stdin
$ fsq scan a_queue echo
args to enqueue

Or from the Python API:

>>> import fsq
>>> for work in fsq.scan('a_queue'):
...     print " ".join(work.arguments)
...     fsq.done('a_queue')
args to enqueue

Work that is successfully completed moves to the done directory:

/var/fsq/a_queue
├── done
|   └── _20131005205643_0_25577_mss_0_args_to_enqueue
├── fail
├── queue
└── tmp

As fsq scans each work item, it obtains an exclusive lock on the work item file, so it is safe to run multiple scan processes (or threads) in parallel on the same queue with no fear of duplicating effort.

Failures in Processing Work

Should work fail during processing:

$ fsq scan a_queue sh -c 'exit 100'

Or from the Python API:

>>> import fsq
>>> for work in fsq.scan('a_queue'):
...     fsq.fail('a_queue')

The failed work will be moved to the fail directory:

/var/fsq/a_queue
├── done
├── fail
|   └── _20131005205643_0_25577_mss_0_args_to_enqueue
├── queue
└── tmp

Work can also fail temporarily, which will cause the work to remain in the queue directory until it is older than FSQ_TTL seconds old, or until it has been tried more than FSQ_MAX_TRIES times unsuccessfully:

$ # exit code 111 indicates temporary failure
$ FSQ_MAX_TRIES=2 fsq scan a_queue sh -c 'exit 100'

Or from the Python API:

>>> import fsq
>>> fsq.set_const('FSQ_MAX_TRIES', 2)
>>> for work in fsq.scan('a_queue'):
...     fsq.fail_tmp(work)

The name of the work item will change to indicate that the item has failed once:

/var/fsq/a_queue
├── done
├── fail
├── queue
|   └── _20131005205643_0_25577_mss_1_args_to_enqueue
└── tmp

Taking Queues Down

To temporaily stop all scanning of any queue, you simply use the fsq down program:

$ fsq down a_queue

Or from the Python API:

>>> import fsq
>>> fsq.down('a_queue')

Which creates a regular file named down in the a_queue directory preventing scan from working on the queue:

/var/fsq/a_queue
├── done
├── down
├── fail
├── queue
|   └── _20131005205643_0_25577_mss_0_args_to_enqueue
└── tmp

To bring a queue back up again, you simply use the fsq up program:

$ fsq up a_queue

Or from the Python API:

>>> import fsq
>>> fsq.up('a_queue')

Which removes the down file, and allows the queue to be scanned properly again.

The tmp Directory

The tmp directory within a_queue is used by fsq under the hood to ensure that all items are enqueued atomically.

The fsq File Name

_20120710213904_0_13044_mss_0_args_to_enqueue
_20120710213904_1_13044_mss_0_args_to_enqueue
+|-----+------| + |-+-| |+| + |------+------|
|      |        |   |    |  |        |
|      |        |   |    |  |        +-> FSQ_DELIMITER seperated
|      |        |   |    |  |            arguments
|      |        |   |    |  +-> tries: number of failed attempts
|      |        |   |    |      to process
|      |        |   |    +-> hostname: the name of the host on
|      |        |   |        which the work-item was enqueued.
|      |        |   +-> pid of the process which enqueued the
|      |        |       work-item
|      |        +-> entropy: should a work-item be generated
|      |            with the same arguments, pid, hostname
|      |            and timestamp, entropy is incremented to
|      |            generate uniqueness.
|      +-> timestamp in FSQ_TIMEFMT format
+-> FSQ_DELIMITER used at enqueue time

ENVIRONMENT

The fsq suite and python library makes use of a number of environment variables (each prefixed by FSQ_), which modify its behavior. Each environment variable is also available as a package-level constant.

Please refer to man 7 fsq for a complete list.

AUTHORS

Matthew Story <matt.story@axial.net>
Isaac (.ike) Levy <ike@blackskyresearch.net>
Will O’Meara <will.omeara@axial.net>

With Additional Contributions From:

Will Martino
Will Slippey
Jacob Yuan

And Thanks To:

William Baxter (For trigger, and for inspiring fsq)
Bruce Guenter (For nullmailer, featuring a simpler file-system queue)
Daniel J Bernstein (For QMail, inspiring trigger and nullmailer)
Release History

Release History

0.2.5

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.3

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
fsq-0.2.5-py27-none-any.whl (96.8 kB) Copy SHA256 Checksum SHA256 2.7 Wheel Apr 7, 2014
fsq-0.2.5.tar.gz (73.1 kB) Copy SHA256 Checksum SHA256 Source Apr 7, 2014

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting