A user space multicast named pipe implementation backed by a regular file
Project description
filebus
A userspace multicast named pipe implementation, backed by a regular file, and accessed via a filebus stdio stream.
Motivation
Imagine a stream of binary data like /dev/urandom
that one wishes
for programs to be able to consume in a multicast fashion. An
obvious solution would be to redirect the binary data to a log file,
and then launch a separate tail
command for each program that needs to
consume output from the log. However, this is not sustainable because
the log size will eventually grow too large if it is unbounded.
In order to solve this problem with filebus
, simply pipe the live
stream to a filebus producer as follows:
strings /dev/urandom | filebus --filename /tmp/urandom.filebus producer
This command causes the file /tmp/urandom.filebus
to behave like a
multicast pipe for filebus consumers which are launched like this:
filebus --filename /tmp/urandom.filebus consumer
Implementation details
The on-disk file is updated via atomic rename while a lock is held. File locking makes it safe for multiple producers to concurrently produce to the same stream.
The filebus --back-pressure
protocol uses deleted files to indicate
consumed buffers. File locking ensures that exactly one
consumer will consume and delete each buffer, and producers will
not attempt to write a new buffer until the previous buffer has been
consumed. A producer writes an empty buffer in order to indicate
EOF, and a consumer will terminate when it reads the empty buffer.
Caveats
The --back-pressure
option implement a lossless protocol, but this
protocol causes only a single consumer to receive a given buffer.
However, it is possible for an unlimited number of consumers which
are not using the --back-pressure
option to eavesdrop on this stream
(provided they have been granted file read permission at the OS level),
albeit in a lossy manner.
In the absence of the --back-pressure
option, consumers will
certainly lose chunks at high data rates, but lower data rates should
be lossless, and consumers should always be able to observe the most
recent chunk if it has not been quickly replaced by another.
Alternative implementations
The bash implementation currently currently reads newline delimited
chunks, whereas the python implementation uses the --sleep-interval
and --block-size
arguments to delimit chunks. The python
implementation is agnostic to the underlying stream in the sense that
it explicitly does not interpret any stream content as a delimiter,
which is a desirable property for filebus, but not essential for many
use cases.
Usage
usage: filebus [-h] [--back-pressure] [--block-size N]
[--impl {bash,python}] [--lossless] [--no-file-monitoring]
[--filename FILE] [--sleep-interval N] [-v]
{producer,consumer} ...
filebus 0.2.0
A user space multicast named pipe implementation backed by a regular file
positional arguments:
{producer,consumer}
producer connect producer side of stream
consumer connect consumer side of stream
optional arguments:
-h, --help show this help message and exit
--back-pressure enable lossless back pressure protocol (unconsumed
chunks cause producers to block)
--block-size N maximum block size in units of bytes
--impl {bash,python}, --implementation {bash,python}
choose an alternative filebus implementation
(alternative implementations interoperate with
eachother)
--lossless an alias for --back-pressure
--no-file-monitoring disable filesystem event monitoring
--filename FILE path of the data file (the producer updates it via
atomic rename)
--sleep-interval N check for new messages at least once every N
seconds
-v, --verbose verbose logging (each occurence increases
verbosity)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file filebus-0.3.1.tar.gz
.
File metadata
- Download URL: filebus-0.3.1.tar.gz
- Upload date:
- Size: 16.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/4.4.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.0 CPython/3.8.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7483be8d259667b16c6834e1ee86f0f81b83dee39cf64b70bbe6abcb06ead9bb |
|
MD5 | ea93dff3dd60c98f633adfa397848e51 |
|
BLAKE2b-256 | 9f6d084d81ee30e8647796777343034a6cfd1580396618de53651c66cc5f3450 |
File details
Details for the file filebus-0.3.1-py3-none-any.whl
.
File metadata
- Download URL: filebus-0.3.1-py3-none-any.whl
- Upload date:
- Size: 21.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/4.4.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.0 CPython/3.8.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | caaa423e81b9e7469a6f5f9f61d05c4d15947888cea3a5edb87eb870293405c0 |
|
MD5 | 51101804d55d14535ea168eef1a2a951 |
|
BLAKE2b-256 | f6168e568f407c782ce2b6498291ff5bd746ea537f714e7150d1499d76f53e43 |