extrainterpreters 0.2a1

Utilities for concurrent code using subinterpreters

extrainterpreters

Utilities to make use of new "interpreters" module in Python 3.12

Status: alpha

Usage: for now one must build cpython 3.12 alpha from the main cpython branch

then, just import "extrainterpreters" and use the Interpreter class as a wrapper to the subinterpreter represented by a single integer handle created by the new interpreters module (that should be in Python stdlib as soon as Eric Snow polishes it)

with an extrainterpreter.Interpreter instance, one can call: inter.run(func_name, *args, **kwargs}) to run code there in the same thread: the return value is just returned as long as it is pickleable. The inter.run_in_thread with the same signature, will start a fresh thead and call the target function there - the Interpreter instance then should be pooled in its "done()" method, and once done, the return value will be available by calling "result".

Also working is a threading like interface:

from extrainterpreters import Interpreter

...

    interp = Interpreter(target=myfunc, args=(...,))
    interp.start()
    ...
    interp.join()

    # bonus: we have return values!

    print(interp.result()

It will even work for methods defined on the REPL, so, just give it a try.

There is a new class in the works which will be able to run code in the sub-interpreter without depending of the PEP554 "run_string" (once the setup is done). All processing in the sub- interpreter will automatically take place in another thread, and the roadmap is towards having a Future object and an InterpreterPoolExecutor compatible with the existing executors in concurrent.Futures

A lot of things here are subject to change. For now, all data exchange takes place in a custom memory buffer with 10MB (it is a bytearray on the parent interpreter, and a memoryview for that on the target)

Data is passed to and back the interpreter using pickle - so, no black magic trying to "kill an object here" and "make it be born there" (not even if I intended to, as ctypes is not currently importable on sub-interpreters).

The good news is that pickle will take care of importing whatever is needed on the other side, so that a function can run.

Roadmap

I plan to get this compatible with concurrent.futures.Executor, and have an easy way to schedule a subinterpreter function execution as an async task.

Also, I should come up with a Queue object to pass data back and forth. As for the data passing mechanism: we will use pickle, no doubt.

When is Beta?

As soon as I get to a well thought way of creating the initial buffer for comunication with a sub-interpreter.

Architecture

The initial implementation used a pre-allocated mmaped file to send/get data from other interpreters, until I found out the mmap call in the sub-interpreter would build a separate mmap object in a different memory region, and a lot of data copying was taking place.

Currently, two C methods retrieve the address of an object implementing the buffer interface, and create a "stand alone" memoryview instance from the address alone. This address is passed to the subinterpreter as a string during the setup stage,