Skip to main content

A user space multicast named pipe implementation backed by a regular file

Project description

filebus

A userspace multicast named pipe implementation, backed by a regular file, and accessed via a filebus stdio stream.

Motivation

Imagine a stream of binary data like /dev/urandom that one wishes for programs to be able to consume in a multicast fashion. An obvious solution would be to redirect the binary data to a log file, and then launch a separate tail command for each program that needs to consume output from the log. However, this is not sustainable because the log size will eventually grow too large if it is unbounded.

In order to solve this problem with filebus, simply pipe the live stream to a filebus producer as follows:

strings /dev/urandom | filebus --filename /tmp/urandom.filebus producer

This command causes the file /tmp/urandom.filebus to behave like a multicast pipe for filebus consumers which are launched like this:

filebus --filename /tmp/urandom.filebus consumer

Implementation details

The on-disk file is updated via atomic rename while a lock is held. File locking makes it safe for multiple producers to concurrently produce to the same stream.

The filebus --back-pressure protocol uses deleted files to indicate consumed buffers. File locking ensures that exactly one consumer will consume and delete each buffer, and producers will not attempt to write a new buffer until the previous buffer has been consumed. A producer writes an empty buffer in order to indicate EOF, and a consumer will terminate when it reads the empty buffer.

Caveats

The --back-pressure option implement a lossless protocol, but this protocol causes only a single consumer to receive a given buffer. However, it is possible for an unlimited number of consumers which are not using the --back-pressure option to eavesdrop on this stream (provided they have been granted file read permission at the OS level), albeit in a lossy manner.

In the absence of the --back-pressure option, consumers will certainly lose chunks at high data rates, but lower data rates should be lossless, and consumers should always be able to observe the most recent chunk if it has not been quickly replaced by another.

Alternative implementations

The bash implementation currently currently reads newline delimited chunks, whereas the python implementation uses the --sleep-interval and --block-size arguments to delimit chunks. The python implementation is agnostic to the underlying stream in the sense that it explicitly does not interpret any stream content as a delimiter, which is a desirable property for filebus, but not essential for many use cases.

Usage

usage: filebus [-h] [--back-pressure] [--block-size N]
               [--impl {bash,python}] [--lossless] [--no-file-monitoring]
               [--filename FILE] [--sleep-interval N] [-v]
               {producer,consumer} ...

  filebus 0.2.0
  A user space multicast named pipe implementation backed by a regular file

positional arguments:
  {producer,consumer}
    producer            connect producer side of stream
    consumer            connect consumer side of stream

optional arguments:
  -h, --help            show this help message and exit
  --back-pressure       enable lossless back pressure protocol (unconsumed
                        chunks cause producers to block)
  --block-size N        maximum block size in units of bytes
  --impl {bash,python}, --implementation {bash,python}
                        choose an alternative filebus implementation
                        (alternative implementations interoperate with
                        eachother)
  --lossless            an alias for --back-pressure
  --no-file-monitoring  disable filesystem event monitoring
  --filename FILE       path of the data file (the producer updates it via
                        atomic rename)
  --sleep-interval N    check for new messages at least once every N
                        seconds
  -v, --verbose         verbose logging (each occurence increases
                        verbosity)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

filebus-0.3.1.tar.gz (16.9 kB view details)

Uploaded Source

Built Distribution

filebus-0.3.1-py3-none-any.whl (21.7 kB view details)

Uploaded Python 3

File details

Details for the file filebus-0.3.1.tar.gz.

File metadata

  • Download URL: filebus-0.3.1.tar.gz
  • Upload date:
  • Size: 16.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.4.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.0 CPython/3.8.5

File hashes

Hashes for filebus-0.3.1.tar.gz
Algorithm Hash digest
SHA256 7483be8d259667b16c6834e1ee86f0f81b83dee39cf64b70bbe6abcb06ead9bb
MD5 ea93dff3dd60c98f633adfa397848e51
BLAKE2b-256 9f6d084d81ee30e8647796777343034a6cfd1580396618de53651c66cc5f3450

See more details on using hashes here.

File details

Details for the file filebus-0.3.1-py3-none-any.whl.

File metadata

  • Download URL: filebus-0.3.1-py3-none-any.whl
  • Upload date:
  • Size: 21.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.4.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.0 CPython/3.8.5

File hashes

Hashes for filebus-0.3.1-py3-none-any.whl
Algorithm Hash digest
SHA256 caaa423e81b9e7469a6f5f9f61d05c4d15947888cea3a5edb87eb870293405c0
MD5 51101804d55d14535ea168eef1a2a951
BLAKE2b-256 f6168e568f407c782ce2b6498291ff5bd746ea537f714e7150d1499d76f53e43

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page