gy-multiprocessing 0.3

Run function in multiple processes

gy-multiprocessing

Installation

via Github

pip install git+https://github.com/guangyu-he/gy-multiprocessing

via PyPI

pip install gy-multiprocessing

Usage

• initializing multiprocessing/multithreading instance
• adding your tasks into the pool, either using loop or sentences
• running the instance

Note:

• the multiprocessing must work in a function or entrance, do not use it barely in the script
• make sure the code does require multiprocessing/multithreading, wrongly using the multiprocessing may even lose performance
• please attention to the queue implementation when using multiprocessing, details check the example below

Examples

Multi Processing

import time
import gy_multiprocessing as gymp


def your_func(a_string: int, queue):
    # NEW from v0.3! if multiprocessing is not used, queue is not a necessary parameter
    # NOTE! you MUST add an argument for queue and use put() method to fetch the returning value if multiprocessing is used

    print(a_string)
    if a_string % 5 == 0:
        time.sleep(2)

    # NEW from v0.3! if multiprocessing is not used, all queue relevant codes can still be kept
    # NEW from v0.3! and the if queue is not None: is not necessary anymore
    # NOTE! if you are missing this method, there will be None result returned for current process if multiprocessing is used
    queue.put(a_string)
    return a_string


if __name__ == '__main__':
    # the multiprocessing must work in a function or entrance
    # do not use it barely

    """
    # initializing the multiprocessing instance
    # the default max_process are your cpu max cores
    # max_process could be infinite, but performance will get suffered when the hardware is overloaded
    """

    # NEW from v0.3! set max_process to 0 will disable multiprocessing
    mp = gymp.MultiProcess(max_process=8)

    # example for multiprocessing in the loop
    outer_loop_times = 5

    # NEW from v0.3! it is possible to test beforehand if the function and args is suitable for multiprocessing
    test_your_func: bool = mp.test(your_func, (1,))

    for current_loop_index in range(outer_loop_times):
        # your running arguments, must be tuple
        args = (current_loop_index,)

        """
        # adding tasks in multiprocessing pool
        """
        mp.add(your_func, args)

    # it is also possible to add task outside the loop
    mp.add(your_func, (10,))

    """
    # running tasks in multiprocessing pool (returned values are optional)
    """
    result = mp.run()
    print(result)

Multi Threads

import gy_multiprocessing as gymp
import time


def your_func(a_string):
    # your single task function

    print(a_string)
    return a_string + "!"


if __name__ == '__main__':
    # the multithreading must work in a function or entrance
    # do not use it barely

    # timing (optional)
    start = time.time()

    """
    # initializing the multithreading instance
    # the default max_threads are your cpu max cores number - 1
    # max_threads can not larger than your cpu max core number
    """
    mt = gymp.MultiThread(max_threads=4)

    # example for multithreading in the loop
    outer_loop_times = 5
    for current_loop_index in range(outer_loop_times):
        args = (str(current_loop_index),)

        """
        # adding tasks in multi threading pool
        """
        mt.add(your_func, args)

    # it is also possible to work without loop
    args = (str(1),)
    mt.add(your_func, args)
    args = (str(2),)
    mt.add(your_func, args)

    """
    # running tasks in multithreading pool (returned values are optional)
    """
    results = mt.run()
    print(results)

    # timing (optional)
    end = time.time() - start
    print("done in {}s".format("%.2f" % end))

Combined Structure

Note: you can not use multiprocessing or sub-multithreading in the multithreading method

If you want to use such structure, based on your needs, considering using sub-multiprocessing or multithreading in multiprocessing structure.

import gy_multiprocessing as gymp


def your_sub_func(b_string: int, queue=None):
    # your function that needs to multithreading/multiprocessing

    b_string += 1
    if queue is not None:
        queue.put(b_string)
    return b_string


def your_mt_func(a_string: int, queue):
    # multithreading in multiprocessing structure

    mt = gymp.MultiThread()
    for current_loop_index in range(a_string):
        # your running arguments, must be tuple
        args = (current_loop_index,)
        mt.add(your_sub_func, args)
    result = mt.run()

    # Do not forget queue!
    queue.put(result)


def your_mp_func(a_string: int, queue):
    # sub-multiprocessing in multiprocessing structure

    smp = gymp.MultiProcess()
    for current_loop_index in range(a_string):
        # your running arguments, must be tuple
        args = (current_loop_index,)
        smp.add(your_sub_func, args)
    result = smp.run()

    # Do not forget queue!
    queue.put(result)


if __name__ == '__main__':

    mp = gymp.MultiProcess()

    outer_loop_times = 10
    for current_loop_index in range(outer_loop_times):
        args = (current_loop_index,)
        mp.add(your_mt_func, args)
    print(mp.run())

    print("\n-----”\n")

    mp = gymp.MultiProcess()

    for current_loop_index in range(outer_loop_times):
        args = (current_loop_index,)
        mp.add(your_mp_func, args)
    print(mp.run())

Updates Log

v0.2.3

bug fix

• fixed an issue casing not adding new process to pool until all processed are done in current pool

v0.2.4

feature

• a silent mode is added to multiprocessing which is possible not showing messages in console
• solved multiprocessing not going to the end when there are internal error exceptions in input function
• solved multiprocessing not going to the end when there are missing queue.put() method in input function

v0.2.4.1 & v0.2.4.2

bug fix

• fixed an issue from readme long desc causing installation error

improvement

• simplified duplicated codes

v0.2.4.3

updates

• package import improvement. PLEASE UPDATE IMPORT AS EXAMPLES ABOVE
• check a boolean type input argument

v0.3

feature

• added a test method to check if the function and args is suitable for multiprocessing
• added a max_process=0 possibility to disable multiprocessing
• queue parameter in user function has a more flexibility to be used among multiprocessing mode on and off
• a new multiprocessing pool logic to check if the subprocess is done

Support

feel free to check source code in GitHub, you are welcome to leave any comments.

2022©Guangyu He, for further support please contact author.
Email: [e-mail]