beelzebub 1.0.0

Lightweight framework for transforming input to output

beelzebub

Beelzebub is a lightweight framework to transform input to output. The base classes aren't meant to be used directly, rather they establish the interfaces of the framework, and provide a basis with which to derive classes for defining a particular transformation workflow.

A workflow consists of a reader class, a writer class, and a processor class. The workflow class instantiates one of each of these classes, and then executes the workflow of reading input from a given source via the reader class, writing output to a given sink via the writer class, and the processor class calls the reader, processes the input, and passes this to the writer.

Both the reader and writer classes are based on a common context manager class. In particular, the open() method can read/write to one of a set of supported iostream types. The iotype must be one of ['file','url','str']. A TypeError exception will be raised otherwise.

The workflow class can optionally set up logging for the workflow (based on the existence of a logger section in the optional configuration dict), and then calls the run() method, passing the source and sink.

As mentioned, an optional configuration dict can be passed when instantiating the workflow object. As a particular workflow will have specific reader, writer and processor classes, the configuration items for each of these components is arbitrary, suited to the particular workflow. However, the framework will look for a toplevel key called reader to pass to the reader class, writer to pass to the writer class, and processor to pass to the processor class. In addition, if a logger key exists, then this will be used to configure logging, via a call to logging.config.dictConfig(conf['logger']).

One of the main uses of the configuration is to specify the iotype for the reader and writer. For example, if the input is read from a file, but the output is to be written to a string, then the configuration should be something like the following:

conf = {'reader': {'iotype': 'file'}, 'writer': {'iotype': 'str'}}
in_file = sys.argv[1]
out_file = None

x = BaseWorkflow(conf=conf)
x.run(in_file, out_file)
print(x.writer.output)

Note that if the output is to be written to a string, then the sink argument (here, out_file) to run() is redundant, and can be set to None. In this case, access the output string via the workflow's writer's output attribute.

If a binary file is to be read or written, then mode should be specified in the corresponding part of the configuration and include the 'b' flag. For example, for reading a binary file from a web server and writing to a local copy:

conf = {
    'reader': {'iotype': 'url', 'mode': 'rb'},
    'writer': {'iotype': 'file', 'mode': 'wb'},
    'logger': {'version': 1, 'loggers': {'beelzebub.base': {'level': 'DEBUG'}}}
}
in_file = 'http://web.server/file.dat'
out_file = './file-copy.dat'

x = BaseWorkflow(conf=conf)
x.run(in_file, out_file)