PEP 554 Subinterpreters usable from Python code

Navigation

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for gwidion from gravatar.com gwidion

Unverified details

These details have not been verified by PyPI
Meta
  • License: GNU Lesser General Public License v3 or later (LGPLv3+)
  • Author: Joao S. O. Bueno
  • Requires: Python >=3.12
  • Provides-Extra: dev
Classifiers
Report project as malware

Project description

extrainterpreters

Utilities to make use of new "interpreters" module in Python 3.12

Status: beta

Usage: Works with cPython >= 3.12

This was made available in 2023 as Python 3.12 came out to allow some experiments and sending objects back and forth subinterpreters.

As of the release of Python 3.14, concurrent.interpreters is made a public API, and is capable of several of the things this project could do - check https://docs.python.org/3.14/library/concurrent.interpreters.html

Right now I am not updating this project - it maybe that I return to it at some point, but really, since free-threading has become am option, I am having a hardtime finding out where a pure Python project could effectively make use of multiple interpreters.

Still, at the eve of Python 3.14, I am issuing an small update so that the API as avaliable up to Python 3.13 is imported from the right place, and the structures in this project work.

Usage

Just import "extrainterpreters" and use the Interpreter class as a wrapper to the subinterpreter represented by a single integer handle created by the new interpreters module:

import extrainterpreters as ET

import time

def my_function():
    time.sleep(1)
    print("running in other interpreter")
    return 42

interp = ET.Interpreter(target=my_function)
interp.start()
print("running in main interpreter")
time.sleep(1.5)
print(f"computation result: {interp.result()}")

history

PEP 554 Describes a Python interface to make use of Sub-interpreters- a long living feature of cPython, but only available through C code embedding Python. While approved, the PEP is not in its final form, and as of Python 3.13 the interpreters module suggested there is made available as _interpreters. (And before that, in Python 3.13, it is available as _xxsubinterpreters.

With the implementation of PEP 684 before Python 3.12, using subinterpreters became a lot more interesting, because each subinterpreter now has an independent GIL. This means that different interpreters (running in different threads) can actually execute Python code in parallel in multiple CPU cores.

"extrainterpreters" offer a nice API to make use of this feature before PEP 554 becomes final, and hopefully, will keep offering value as a nice API wrapper when it is final. Enjoy!

forms of use

with an extrainterpreter.Interpreter instance, one can call: inter.run(func_name, *args, **kwargs}) to run code there in the same thread: the return value is just returned as long as it is pickleable. The inter.run_in_thread with the same signature, will start a fresh thead and call the target function there - the Interpreter instance then should be pooled in its "done()" method, and once done, the return value will be available by calling "result".

Also working is a threading like interface:

from extrainterpreters import Interpreter

...

    interp = Interpreter(target=myfunc, args=(...,))
    interp.start()
    ...
    interp.join()

    # bonus: we have return values!

    print(interp.result()

It will even work for methods defined on the REPL, so, just give it a try.

There is a new class in the works which will be able to run code in the sub-interpreter without depending of the PEP554 "run_string" (once the setup is done). All processing in the sub- interpreter will automatically take place in another thread, and the roadmap is towards having a Future object and an InterpreterPoolExecutor compatible with the existing executors in concurrent.Futures

A lot of things here are subject to change. For now, all data exchange takes place in a custom memory buffer with 10MB (it is a bytearray on the parent interpreter, and a memoryview for that on the target)

Data is passed to and back the interpreter using pickle - so, no black magic trying to "kill an object here" and "make it be born there" (not even if I intended to, as ctypes is not currently importable on sub-interpreters).

The good news is that pickle will take care of importing whatever is needed on the other side, so that a function can run.

API and Usage:

In beta stage the suggestion is to use this as one would threading.Thread - see the example above - and do not rely on the provided Queue class (seriously it is broken right now), until some notice about it is given.

Roadmap

first the basics:

we should get Lock, Rlock, Queue and some of the concurrency primitives
as existing for threads, async and subprocessing working - at that point
this should be production quality

second, the bells and whistles

I plan to get this compatible with concurrent.futures.Executor, and have an easy way to schedule a subinterpreter function execution as an async task.

Also, I should come up with a Queue object to pass data back and forth. As for the data passing mechanism: we will use pickle, no doubt.

Architecture

The initial implementation used a pre-allocated mmaped file to send/get data from other interpreters, until I found out the mmap call in the sub-interpreter would build a separate mmap object in a different memory region, and a lot of data copying was taking place.

Currently, two C methods retrieve the address of an object implementing the buffer interface, and create a "stand alone" memoryview instance from the address alone. This address is passed to the subinterpreter as a string during the setup stage,

Project details

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for gwidion from gravatar.com gwidion

Unverified details

These details have not been verified by PyPI
Meta
  • License: GNU Lesser General Public License v3 or later (LGPLv3+)
  • Author: Joao S. O. Bueno
  • Requires: Python >=3.12
  • Provides-Extra: dev
Classifiers

Release history Release notifications | RSS feed

This version

0.3.0

0.2.0

0.2a1 pre-release

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

extrainterpreters-0.3.0.tar.gz (38.8 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

extrainterpreters-0.3.0-cp313-cp313-musllinux_1_2_x86_64.whl (45.1 kB view details)

Uploaded CPython 3.13musllinux: musl 1.2+ x86-64

extrainterpreters-0.3.0-cp313-cp313-musllinux_1_2_i686.whl (45.0 kB view details)

Uploaded CPython 3.13musllinux: musl 1.2+ i686

extrainterpreters-0.3.0-cp313-cp313-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (44.9 kB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ x86-64manylinux: glibc 2.5+ x86-64

extrainterpreters-0.3.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl (44.5 kB view details)

Uploaded CPython 3.13manylinux: glibc 2.17+ i686manylinux: glibc 2.5+ i686

extrainterpreters-0.3.0-cp313-cp313-macosx_11_0_arm64.whl (38.3 kB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

extrainterpreters-0.3.0-cp313-cp313-macosx_10_13_x86_64.whl (37.9 kB view details)

Uploaded CPython 3.13macOS 10.13+ x86-64

extrainterpreters-0.3.0-cp312-cp312-musllinux_1_2_x86_64.whl (45.0 kB view details)

Uploaded CPython 3.12musllinux: musl 1.2+ x86-64

extrainterpreters-0.3.0-cp312-cp312-musllinux_1_2_i686.whl (44.9 kB view details)

Uploaded CPython 3.12musllinux: musl 1.2+ i686

extrainterpreters-0.3.0-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (44.9 kB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ x86-64manylinux: glibc 2.5+ x86-64

extrainterpreters-0.3.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl (44.5 kB view details)

Uploaded CPython 3.12manylinux: glibc 2.17+ i686manylinux: glibc 2.5+ i686

extrainterpreters-0.3.0-cp312-cp312-macosx_11_0_arm64.whl (38.3 kB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

extrainterpreters-0.3.0-cp312-cp312-macosx_10_13_x86_64.whl (37.8 kB view details)

Uploaded CPython 3.12macOS 10.13+ x86-64

File details

Details for the file extrainterpreters-0.3.0.tar.gz.

File metadata

  • Download URL: extrainterpreters-0.3.0.tar.gz
  • Upload date:
  • Size: 38.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for extrainterpreters-0.3.0.tar.gz
Algorithm Hash digest
SHA256 2ae19ca2564ae3ea64d30ae3832ebc9de6a3bea01e4978432aac059988b3e8d9
MD5 0f9fd29245fb2cd9f80abab44d041d2d
BLAKE2b-256 04c4bd7bcc5c9ec60137e9e3590d357c3b161b4bc979455ca2d036dbb1729300

See more details on using hashes here.

Provenance

The following attestation bundles were made for extrainterpreters-0.3.0.tar.gz:

Publisher: pypi.yml on jsbueno/extrainterpreters

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file extrainterpreters-0.3.0-cp313-cp313-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for extrainterpreters-0.3.0-cp313-cp313-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 ecf6672839f56454a27a9f49f4b82d840662fd6887aa05e1e46dce3c082b87d5
MD5 61787bed25fad51ff9009a6aa726f4c3
BLAKE2b-256 e068baaff733336da2a74f49fa9a68b6418e9df35ff7f14fb08eb66dc9db88eb

See more details on using hashes here.

Provenance

The following attestation bundles were made for extrainterpreters-0.3.0-cp313-cp313-musllinux_1_2_x86_64.whl:

Publisher: pypi.yml on jsbueno/extrainterpreters

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file extrainterpreters-0.3.0-cp313-cp313-musllinux_1_2_i686.whl.

File metadata

File hashes

Hashes for extrainterpreters-0.3.0-cp313-cp313-musllinux_1_2_i686.whl
Algorithm Hash digest
SHA256 1870c7adcd4c9273610da46319e06b64e723e00cb9e216e6661e19a05b53588f
MD5 f842b98450448230461084fd686df65b
BLAKE2b-256 27d380dd1f911f67834f9c2629cdc695545b952e94e62ddd3be4e721b4b1d9dc

See more details on using hashes here.

Provenance

The following attestation bundles were made for extrainterpreters-0.3.0-cp313-cp313-musllinux_1_2_i686.whl:

Publisher: pypi.yml on jsbueno/extrainterpreters

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file extrainterpreters-0.3.0-cp313-cp313-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for extrainterpreters-0.3.0-cp313-cp313-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 b42f1b221a87797221f978baaeba24a49adc35f1d70c253a985121a879ef6d2b
MD5 ba53722e9c3519566d98c9bf0196e79f
BLAKE2b-256 7d8000a328496794c2cb145b4084e0dab356cceb1fa8f94106a1f486d3924ac0

See more details on using hashes here.

Provenance

The following attestation bundles were made for extrainterpreters-0.3.0-cp313-cp313-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: pypi.yml on jsbueno/extrainterpreters

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file extrainterpreters-0.3.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl.

File metadata

File hashes

Hashes for extrainterpreters-0.3.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm Hash digest
SHA256 3ed15f31d775b0f1cc9bc9ed1147f52b1197007b42edf991b2bd95035097c1ed
MD5 074eee2858a0e564958a259a7b1cd6f6
BLAKE2b-256 f859dd490d6a9a70bcc9ec977a247c7d02a91cf97ff423c33dcac29e471839c7

See more details on using hashes here.

Provenance

The following attestation bundles were made for extrainterpreters-0.3.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl:

Publisher: pypi.yml on jsbueno/extrainterpreters

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file extrainterpreters-0.3.0-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for extrainterpreters-0.3.0-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 018c4a2d7db0ee052f19292bc3332f82630f1ca6ee6679b6544dea19df6a3e1a
MD5 8bdd3dd6d3d4241cfd5013d0615412b7
BLAKE2b-256 b5082ec9cd533816a277953236769006935406632cf900b2333874e9b08a226f

See more details on using hashes here.

Provenance

The following attestation bundles were made for extrainterpreters-0.3.0-cp313-cp313-macosx_11_0_arm64.whl:

Publisher: pypi.yml on jsbueno/extrainterpreters

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file extrainterpreters-0.3.0-cp313-cp313-macosx_10_13_x86_64.whl.

File metadata

File hashes

Hashes for extrainterpreters-0.3.0-cp313-cp313-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 44323a1df918eba1b703f65adb819c2e2712401cd47a47a529448f84aa428e12
MD5 0d80684085dd12347c54d4f6080144d0
BLAKE2b-256 6c76c15d56d18b96f59dac430ddaa7251793ba9060b8cba1adb6a7968c2c088e

See more details on using hashes here.

Provenance

The following attestation bundles were made for extrainterpreters-0.3.0-cp313-cp313-macosx_10_13_x86_64.whl:

Publisher: pypi.yml on jsbueno/extrainterpreters

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file extrainterpreters-0.3.0-cp312-cp312-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for extrainterpreters-0.3.0-cp312-cp312-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 81ace9374969e811b320c7f4a7687a7025dc028da6ec673a7aaebcab74f46559
MD5 378908ac9da6e310defe67f34b421bca
BLAKE2b-256 de058bf512d3f94a72b34ccb4731b430a8c7bfc598af137a5760dfeed9de31ea

See more details on using hashes here.

Provenance

The following attestation bundles were made for extrainterpreters-0.3.0-cp312-cp312-musllinux_1_2_x86_64.whl:

Publisher: pypi.yml on jsbueno/extrainterpreters

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file extrainterpreters-0.3.0-cp312-cp312-musllinux_1_2_i686.whl.

File metadata

File hashes

Hashes for extrainterpreters-0.3.0-cp312-cp312-musllinux_1_2_i686.whl
Algorithm Hash digest
SHA256 e9350da419c32e869d9d0e66fbea1eaa3f36291992ed0ea8d5e06b48f2b2f2e4
MD5 931ba03746bdad0d5908e4138d94747e
BLAKE2b-256 a24731747d71bf59f073af2f1cd84beaf3042e5fc40a93c90ee8fde69d124c9b

See more details on using hashes here.

Provenance

The following attestation bundles were made for extrainterpreters-0.3.0-cp312-cp312-musllinux_1_2_i686.whl:

Publisher: pypi.yml on jsbueno/extrainterpreters

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file extrainterpreters-0.3.0-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for extrainterpreters-0.3.0-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 2f29abc9d05d419e458b39dc6a1e2b127f2acbfcb77d54b178a242bfb5908535
MD5 e4e85e0d89df834535f0b0aac1b72e57
BLAKE2b-256 5068affe17d507a7f574f43b70948aa23a54a08aa52f82be5bd87366dab4a09e

See more details on using hashes here.

Provenance

The following attestation bundles were made for extrainterpreters-0.3.0-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: pypi.yml on jsbueno/extrainterpreters

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file extrainterpreters-0.3.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl.

File metadata

File hashes

Hashes for extrainterpreters-0.3.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm Hash digest
SHA256 36cca3cea8aeeca0838291771b1793af7ace2603f940614c42fdd9e096436c58
MD5 958b6613f1cdb3663134e0acac8d4b81
BLAKE2b-256 c70fa5359cd020911736e8577316b99d2e3b34e5407c0dfa96c7d4864301508b

See more details on using hashes here.

Provenance

The following attestation bundles were made for extrainterpreters-0.3.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl:

Publisher: pypi.yml on jsbueno/extrainterpreters

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file extrainterpreters-0.3.0-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for extrainterpreters-0.3.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 4b938eb0e8d5097bca484c08e7425027fac325436031d4d02b50f046650fe423
MD5 7f571e6a65d672615035f5473ca8a21f
BLAKE2b-256 2a6fba498f49c1456ba2b6101a73402e3b253f0901efdd09788acf8b64e9f063

See more details on using hashes here.

Provenance

The following attestation bundles were made for extrainterpreters-0.3.0-cp312-cp312-macosx_11_0_arm64.whl:

Publisher: pypi.yml on jsbueno/extrainterpreters

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file extrainterpreters-0.3.0-cp312-cp312-macosx_10_13_x86_64.whl.

File metadata

File hashes

Hashes for extrainterpreters-0.3.0-cp312-cp312-macosx_10_13_x86_64.whl
Algorithm Hash digest
SHA256 aa63e29f9725bc18f23cf4c68e2feb3164e38ece364d41e5aef4e04fbe47e879
MD5 bb435bb22121c41085eee1e84aceb32f
BLAKE2b-256 398cec3462ed179fa2f3a9dc7f1279255d890a00e3a0f168fa9c2cb782e81c5c

See more details on using hashes here.

Provenance

The following attestation bundles were made for extrainterpreters-0.3.0-cp312-cp312-macosx_10_13_x86_64.whl:

Publisher: pypi.yml on jsbueno/extrainterpreters

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page