FemtoQueue

Ever wanted a zero-dependency, filesystem-backed, lock-free, durable, concurrent, retrying task queue implementation? No?

Note: This is pre-release software. Backwards compatibility will be guaranteed after v1.0.

Example

from femtoqueue import FemtoQueue

q = FemtoQueue(data_dir = "fq", node_id = "node1")
q.push("foobar".encode("utf-8"))

while task := q.pop():
    # Do something with `task.data`
    q.done(task) # or q.fail(task)

print("All tasks processed")

Installation

femtoqueue is available on PyPI.

uv add femtoqueue # using uv
pip install femtoqueue # using pip

Or just chuck the femtoqueue.py file into your Python 3 project. There are no dependencies other than the standard library.

Features

This mini-library provides the FemtoQueue class with the standard queue interface, with some additions:

Each task corresponds to one file in the data_dir directory. State changes are atomic since they use rename(). The task can contain whatever you want, the queue does not inspect it in any way.

Each concurrent worker node (library user) must have a stable identifier node_id. This way workers can automatically retry a task if they unexpectedly crash in the middle of processing.

Scheduled tasks are moved to the back of the pending queue when the specified wall-clock time is less than the current wall-clock time. Stale tasks (i.e. in progress for too long) are moved back to pending automatically after a timeout is reached (default: 30s). These events are processed during a pop() call; the library does not run any background jobs by itself.

Tasks are ordered using the system-provided monotonic clock to avoid issues such as NTP-related clock skew or daylight savings time moving the clock backwards in time, which would be possible if ordering was based on wall-clock time. As long as all worker nodes have the same clock source, pending tasks are guaranteed to be processed in insertion order, with microsecond precision.

But isn't this slow?

I wouldn't migrate away from your production queue system just yet, but this is faster than you'd expect. Easily fast enough for some small or medium project. Turns out, creating and renaming files is pretty snappy.

Running the microbenchmark python benchmark_mini.py on a Macbook Pro M1 reports around 4500 pushed tasks/sec and 400 popped tasks/sec. Most of the time is spent opening files. A heftier FreeBSD machine was able to reach 21000 pushed/sec and 7400 popped/sec. Note that these are not very scientific numbers.

Unit tests

python test.py

Author and license

Jan Tuomi <jan@jantuomi.fi>. Licensed under Apache-2.0. All rights reserved.