mp4ansi

A simple ANSI-based terminal that provides various capabilities for showing off and/or scaling out your programs execution. MP4ansi is an abstraction of multiprocessing that leverages both the Terminal and ProgressBar. See the examples for more detail.

Installation

pip install mp4ansi

Examples

Various examples are included to demonstrate the mp4ansi package. To run the examples, build the Docker image and run the Docker container using the instructions described in the Development section.

MP4ansi

MP4ansi will scale execution of a specified function across multiple background processes, where each process is mapped to specific line on the terminal. As the function executes its log messages will automatically be written to the respective line on the terminal. The number of processes along with the arguments to provide each process is specified as a list of dictionaries. The number of elements in the list will dictate the total number of processes to execute (as well as the number of lines in the terminal). The result of each function is written to the respective dictionary element and can be interogated upon completion.

MP4ansi is a subclass of mpmq, see the mpmq for more information.

Here is a simple example:

from mp4ansi import MP4ansi
import random, names, logging
logger = logging.getLogger(__name__)

def do_work(*args):
    total = random.randint(50, 100)
    logger.debug(f'processing total of {total}')
    for _ in range(total):
        logger.debug(f'processed {names.get_full_name()}')
    return total

def main():
    process_data = [{} for item in range(8)]
    print('Procesing names...')
    MP4ansi(function=do_work, process_data=process_data).execute()
    print(f"Total names processed {sum([item['result'] for item in process_data])}")

if __name__ == '__main__':
    main()

Executing the code above (example1) results in the following:

Note the function being executed do_work has no context about multiprocessing or the terminal; it simply perform a function on a given dataset. MP4ansi takes care of setting up the multiprocessing, setting up the terminal, and maintaining the thread-safe queues that are required for inter-process communication.

Let's update the example to add a custom identifer for each process and to show execution as a progress bar. To do this we need to provide additonal configuration via the optional config parameter. Configuration is supplied as a dictionary; id_regex instructs how to query for the identifer from the log messages. For the progress bar, we need to specify total and count_regex to instruct how to query for the total and for when to count that an item has been processed. The value for these settings are specified as regular expressions and will match the function log messages, thus we need to ensure our function has log statements for these. If each instance of your function executes on a static data range then you can specify total as an int, but in this example the data range is dynamic, i.e. each process will execute on varying data ranges.

from mp4ansi import MP4ansi
import names, random, logging
logger = logging.getLogger(__name__)

def do_work(*args):
    logger.debug(f'processor is {names.get_last_name()}')
    total = random.randint(50, 125)
    logger.debug(f'processing total of {total}')
    for _ in range(total):
        logger.debug(f'processed {names.get_full_name()}')
    return total

def main():
    process_data = [{} for item in range(8)]
    config = {
        'id_regex': r'^processor is (?P<value>.*)$',
        'progress_bar': {
            'total': r'^processing total of (?P<value>\d+)$',
            'count_regex': r'^processed (?P<value>.*)$',
            'progress_message': 'Finished processing names'}}
    print('Procesing names...')
    MP4ansi(function=do_work, process_data=process_data, config=config).execute()
    print(f"Total names processed {sum([item['result'] for item in process_data])}")

if __name__ == '__main__':
    main()

Executing the code above (example2) results in the following:

Terminal

The package also exposes a Terminal class if you wish to consume the terminal capabilities without executing background processes. Here is an example for how to do that:

from mp4ansi import Terminal
from essential_generators import DocumentGenerator
import time, random

def main():
    print('generating random sentences...')
    count = 15
    docgen = DocumentGenerator()
    terminal = Terminal(count)
    terminal.write_lines()
    terminal.hide_cursor()
    for _ in range(800):
        index = random.randint(0, count - 1)
        terminal.write_line(index, docgen.sentence())
        time.sleep(.01)
    terminal.write_lines(force=True)
    terminal.show_cursor()

if __name__ == '__main__':
    main()

Executing the code above (example5) results in the following:

Development

Clone the repository and ensure the latest version of Docker is installed on your development server.

Build the Docker image:

docker image build \
-t \
mp4ansi:latest .

Run the Docker container:

docker container run \
--rm \
-it \
-v $PWD:/code \
mp4ansi:latest \
/bin/bash

Execute the build:

pyb -X