gy-multiprocessing 0.2.4

Run function in multiple processes

gy-multiprocessing

Installation

via Github

pip install git+https://github.com/guangyu-he/gy-multiprocessing

via PyPI

pip install gy-multiprocessing

Usage

• initializing multiprocessing/multithreading instance
• adding your tasks into the pool, either using loop or sentences
• running the instance

Note:

• the multiprocessing must work in a function or entrance, do not use it barely in the script
• make sure the code does require multiprocessing/multithreading, wrongly using the multiprocessing may even lose performance
• please attention to the queue implementation when using multiprocessing, details check the example below

Examples

Multi Processing

import gy_multiprocessing.multiprocessing.multi_process as gymp
import time


def your_func(a_string: int, queue):
    # NOTE! you MUST add an argument for queue and use put() method to fetch the returning value

    print(a_string)
    if a_string % 5 == 0:
        time.sleep(2)

    # NOTE! if you are missing this method, there will be None result returned for current process
    queue.put(a_string)


if __name__ == '__main__':
    # the multiprocessing must work in a function or entrance
    # do not use it barely

    """
    # initializing the multiprocessing instance
    # the default max_process are your cpu max cores
    # max_process could be infinite, but performance will get suffered when the hardware is overloaded
    """
    mp = gymp.MultiProcess(max_process=8)

    # example for multiprocessing in the loop
    outer_loop_times = 5
    for current_loop_index in range(outer_loop_times):
        # your running arguments, must be tuple
        args = (current_loop_index,)

        """
        # adding tasks in multiprocessing pool
        """
        mp.add(your_func, args)

    # it is also possible to add task outside the loop
    mp.add(your_func, (10,))

    """
    # running tasks in multiprocessing pool (returned values are optional)
    """
    result = mp.run()
    print(result)

Multi Threads

import gy_multiprocessing.multithreading.multi_thread as gymt
import time


def your_func(a_string):
    # your single task function

    print(a_string)
    return a_string + "!"


if __name__ == '__main__':
    # the multithreading must work in a function or entrance
    # do not use it barely

    # timing (optional)
    start = time.time()

    """
    # initializing the multithreading instance
    # the default max_threads are your cpu max cores number - 1
    # max_threads can not larger than your cpu max core number
    """
    mt = gymt.MultiThread(max_threads=4)

    # example for multithreading in the loop
    outer_loop_times = 5
    for current_loop_index in range(outer_loop_times):
        args = (str(current_loop_index),)

        """
        # adding tasks in multi threading pool
        """
        mt.add(your_func, args)

    # it is also possible to work without loop
    args = (str(1),)
    mt.add(your_func, args)
    args = (str(2),)
    mt.add(your_func, args)

    """
    # running tasks in multithreading pool (returned values are optional)
    """
    results = mt.run()
    print(results)

    # timing (optional)
    end = time.time() - start
    print("done in {}s".format("%.2f" % end))

Combined Structure

Note: you can not use multiprocessing or sub-multithreading in the multithreading method

If you want to use such structure, based on your needs, considering using sub-multiprocessing or multithreading in multiprocessing structure.

import gy_multiprocessing.multiprocessing.multi_process as gymp
import gy_multiprocessing.multithreading.multi_thread as gymt


def your_sub_func(b_string: int, queue=None):
    # your function that needs to multithreading/multiprocessing

    b_string += 1
    if queue is not None:
        queue.put(b_string)
    return b_string


def your_mt_func(a_string: int, queue):
    # multithreading in multiprocessing structure

    mt = gymt.MultiThread()
    for current_loop_index in range(a_string):
        # your running arguments, must be tuple
        args = (current_loop_index,)
        mt.add(your_sub_func, args)
    result = mt.run()

    # Do not forget queue!
    queue.put(result)


def your_mp_func(a_string: int, queue):
    # sub-multiprocessing in multiprocessing structure

    smp = gymp.MultiProcess()
    for current_loop_index in range(a_string):
        # your running arguments, must be tuple
        args = (current_loop_index,)
        smp.add(your_sub_func, args)
    result = smp.run()

    # Do not forget queue!
    queue.put(result)


if __name__ == '__main__':

    mp = gymp.MultiProcess()

    outer_loop_times = 10
    for current_loop_index in range(outer_loop_times):
        args = (current_loop_index,)
        mp.add(your_mt_func, args)
    print(mp.run())

    print("\n-----”\n")

    mp = gymp.MultiProcess()

    for current_loop_index in range(outer_loop_times):
        args = (current_loop_index,)
        mp.add(your_mp_func, args)
    print(mp.run())

Updates Log

v0.2.3

bug fix

• fixed an issue casing not adding new process to pool until all processed are done in current pool

feel free to check source code in GitHub, you are welcome to leave any comments.

v0.2.4

feature

• a silent mode is added to multiprocessing which is possible not showing messages in console
• solved multiprocessing not going to the end when there are internal error exceptions in input function
• solved multiprocessing not going to the end when there are missing queue.put() method in input function

improvement

• simplified duplicated codes

2022©Guangyu He, for further support please contact author.
Email: [e-mail]