streamcapture 0.1.0

Capture output streams such as sys.stdout and sys.stderr.

streamcapture: capture the outputs of Python streams, such as sys.stdout and sys.stderr

Author: Sébastien Loisel

Installation

pip install streamcapture

Example usage

import streamcapture, sys, os
print("This does not get saved to the log file")
with streamcapture.StreamCapture(sys.stdout,open('logfile.txt','wb')):
        os.write(sys.stdout.fileno(),b"Hello, captured world!\n")
        os.system('echo Hello from the shell')
        print("More capturing")
print("This also does not get saved to the log file")

After execution, this will create a file logfile.txt in the current directory, containing the relevant captured outputs.

Documentation

Class StreamCapture(stream, writer, echo=True, monkeypatch=None) is able to capture, at the operating system level, the data being written to the given stream. A typical use is to capture all outputs to sys.stdout and sys.stderr and log them to a file. This will even capture the outputs of spawned shell commands.

StreamCapture works by essentially using os.dup2 to send stream.fileno() to a os.pipe(). A separate thread is used to listen to that os.pipe and send the outputs to the destination writer stream. StreamCapture also uses os.dup to duplicate the original filedescriptor to be able to restore it at the end. This duplicated filedescriptor is stored in StreamCapture.dup_fd, and writing to this filedescriptor results in writing to the original file, before it was redirected. For example, when redirecting sys.stdout, one can still write to the terminal by writing directly to StreamCapture.dup_fd with os.write().

On Windows, sys.stdout and sys.stderr do not take kindly to their fileno() being redirected with os.dup2. StreamCapture features an optional workaround, enabled by the monkeypatching optional parameter to the constructor. When enabled, the workaround overwrites stream.write(...) by an implementation that sends everything to os.write(self.fd,...). This workaround is enabled when monkeypatching=True and disabled when monkeypatching=False. The default is monkeypatching=None, in which case monkeypatching is enabled only when platform.system()=='Windows'.

When writing to multiple streams and file descriptors, sometimes the order in which the writes appear can be surprising. For example, when writing to stderr and stdout, these outputs do not necessarily appear in the order in which they occurred during the program execution, because of various levels of buffering that occur in Python, the C library or the operating system.

At the Python level, streams can be flush()ed to attempt to reduce the delay before a write() has delivered its payload. Furthermore, os.fsync() can be used on some, but not all, file descriptors. However, os.fsync() usually causes an exception if it is called on sys.stdout.fileno() or on a os.pipe(). In principle, the operating system should promtly flush any buffers when a file descriptor is os.close()d, but there is no guarantee. To complicate matters, although one usually prefers minimal buffering for outputs that go to the console, Python tries very hard to force some sort of buffering on text-mode files.

We have tried to prevent most forms of buffering at the Python level and at the operating system levels, but when multiple file descriptors are used, or at the boundary when a StreamCapture starts or stops capturing the underlying stream, some outputs that go to the console may appear in an unexpected order.

More sophisticated behaviors can be handled by implementing a custom stream-like object. The writer object should implement functions writer.write(data), where data is a byte string, and writer.close().

The echo flag can be set at construction time StreamCapture(...,echo=True) and defaults to True. In this mode, all captured outputs are sent both to the writer and also to StreamCapture.dup_fd. This allows one to send, e.g. stdout to a log file while simultaneously printing it to the console, similar to the tee console command in Unix. The echo flag can be set to False to disable this.

One can call StreamCapture.close() to cleanly unwind the captured streams. This is automatically done if StreamCapture is used in a with block.