cm-shared-generator 0.1.4

Shared generator between process

Python library : cm_shared_generator

Resume

In python, it exist a generator type which can be really useful.

The cm_shared_generator library provide a Shared_Generator which share generator between thread and process safely.

Install

Pre-required

Global requirements

• Python 3 (sudo apt update && sudo apt install python3 -y)

Install requirements

• Setuptools (sudo apt update && sudo apt install python3-pip -y && pip3 install setuptools)

Installation

python3 setup.py install

Utilization

The utilization of the Shared_Generator is quite simple. Its prototype is like :

Shared_Generator(self, arg_generator, DEFAULT_GENERATOR_TRAITMENT_FUNCTION=None, FORCE=False, MAX_OBJECT_IN_QUEUE=len(os.sched_getaffinity(0)), TIMEOUT=None)

Its parameters are defined as :

• Required :
  • arg_generator -> The generator to shared
• Optional :
  • DEFAULT_GENERATOR_TRAITMENT_FUNCTION (Default : None) -> Function to treat default type of value from the generator. Possible value :
    • Shared_Generator.DEFAULT_GENERATOR_TRAITMENT_CLASS (Create a deep copy of the class before the insertion into the Queue).
    • Your own function !
  • FORCE (Default : False) -> If True, the generator will be defined even his type is not a proper generator.
  • MAX_OBJECT_IN_QUEUE (Default : Nb of logical processors) : Defined the maximum of elements in the queue at the same time. ( > 0 )
  • NUMBER_PUT_RETRY (Default : None) -> Number of retry to acquire the semaphore which define if the generator process can continue to fill the Generator list before quitting timeout.
  • TIMEOUT (Default : Shared_Generator.DEFAULT_TIMEOUT) : Defined the timeout of Semaphores (The Shared_Generator will be stopped after (1+NUMBER_PUT_RETRY)*TIMEOUT if no process has taken a value from the generator.

Example :

from cm_shared_generator import Shared_Generator

# Simple integer generator
def int_generator():
    i = 0
    while i<10:
        i+=1
        yield i
    return

my_shared_generator = Shared_Generator(int_generator, MAX_OBJECT_IN_QUEUE=2) # Creation of the generator with a Maximum lenght queue of 2

my_shared_generator.start(NEW_PROCESS=True) # Launch the generator in a new process

val = my_shared_generator.next() # Get the next value of the generator

my_shared_generator.stop() # Stop the generator

For more examples, check unit tests in tests directory.