Skip to main content

Framework for combining different python interpeters

Project description

PyPI Package Code Health Test Health Code Coverage

Multi-intepreter execution environment

cpy2py allows multiple interpreters to act as one application. In parallel to the main interpreter, other interpreters are run to execute parts of the application.

Quick Guide

Twinterpreters and TwinMasters

A twinterpreter is simply another interpreter running as a subprocess - with some glue and magic sprinkled on it. You can control and create them using a cpy2py.TwinMaster.

You should only ever worry about two methods: TwinMaster.start launches the twinterpreter. TwinMaster.execute executes an arbitrary callable in the twinterpreter.

from cpy2py import TwinMaster
from my_module import my_function
twinterpreter = TwinMaster('pypy')
twinterpreter.start()

if __name__ == "__main__":
    twinterpreter.execute(my_function, 1, 2, 3, 'ka-pow!', doctor="who?")

TwinObjects

The real power of cpy2py are Twins - objects living in one twinterpreter and being represented by proxies in any other interpeter. Using twins, you can seamlessly split your application across multiple twinterpreters.

You create twins by inheriting from cpy2py.TwinObject instead of object and setting a __twin_id__. That’s it.

from cpy2py import TwinObject
class SuperComputer(TwinObject):
    __twin_id__ = 'pypy'  # makes class native to pypy twinterpeter

    def megaloop(self, x, y):
        return sum(a+b for a in range(x) for b in range(y))

class CWrapper(TwinObject):
    __twin_id__ = 'python'  # makes class native to python twinterpeter

    def callme(self, who, what="buy milk"):
        return some_clib.c_fcn_cll_cplx_xmpl(who, what)

If you don’t set __twin_id__ on a child of cpy2py.TwinObject, the class will always be native to the main interpreter. Handy for all the stuff that’s needed everywhere but really doesn’t belong anywhere.

note:

At the moment, you have to explicitly start a class’s native twinterpreter before instantiating the class. Only the main interpreter is always available, of course.

TwinFunctions

Instead of full-fletched objects, you can also define functions as twins. These are automatically called in their native twinterpreter.

from cpy2py import twinfunction

@twinfunction('pypy')
def superlooper(count=1000, add=3, start=0):
    for _ in range(count):
        start += add
    return add

print(superlooper(int(1E6), 1))
note:

A cpy2py.twinfunction is a regular function wrapping a callable. Unlike a cpy2py.TwinObject, it will not pass attribute assignments.

Debugging

The core of cpy2py supports some logging facilities. All such loggers are children of the __cpy2py__ logger. By default, no active handlers are attached and propagation is disabled. If needed, you reconfigure them like any other logging logger to suit your needs. Note that if python is run with the -O flag, several logging calls are skipped entirely to improve performance.

For small scale debugging, one can set the environment variable CPY2PY_DEBUG. If it is defined and not empty, logging output is written to stderr. In addition, if it names a valid logging level, that logging level is used.

Note that loggers are meant for development and only address the internal state. Your application should not depend on this information. Unless cpy2py misbehaves (or you suspect it to), ignore its logging.

Current Status

CPy2Py is stable at its core, but still has some features missing. What’s there is more than sufficient to significantly enhance your applications.

Features

  • Seamlessly integrates into python code.

    • All internals are wrapped away behind the plain python interfaces. No eval, exec or code strings required.

    • Lightweight hooks optimize objects and functions for use with cpy2py.

    • If needed, any pickle’able callable can be dispatched to another interpreter.

  • Objects natively integrate with twinterpreters.

    • Objects can live in a specific interpreter, with proxies replacing them in others. Classes and instances transparently interact with cpy2py in the background.

    • Both class and instance attributes work as expected. Methods, classmethods, staticmethods and descriptors are fully supported.

    • Inheritance is fully supported, including multiple inheritance. Affiliation to interpreters can be changed freely.

  • A wide range of interpeters is supported.

    • Pure python, no dependencies means perfect portability.

    • Any interpreter compatible with python 2.6 to 3.7 is supported.

    • Virtual Environments work out of the box.

    • Tested with cpython and pypy, on Linux and Mac OSX.

Gotchas/Limitations

  • Importing functions and classes from __main__ may fail if the module can only be imported via its path.

  • By default, calls across interpreters are blocking and not threadsafe. If recursion switches between twinterpreters, cpy2py.TwinMaster must use the 'async' kernel.

  • Module level settings are not synchronized. For example, configuration of logging is not applied to twinterpreters. Use cpy2py.twinterpreter.group_state.TwinGroupState for initialisation, write modules aware of twinterpreters, or use immutable module-level initializers.

  • A weakref to objects only takes local references into account, not cross-interpreter references.

Performance

Dispatching to another twinterpreter adds about 200 - 300 us of overhead. This is mainly due to serialization for the IPC between the interpreters. Using the asynchronous kernel, there is an additional overhead for creating threads.

In general, twinterpreters get faster the shorter they have to wait between requests. pypy twinterpreters benefit from a high number of requests, allowing their JIT to warm up. Python3 connections are the fastest, provided that both twinterpreters support pickle protocol 4.

A notable fraction of time is spent on debugging output via logging. Even if no output is produced, cpy2py is optimized to a point where the logging call is noticeable. If needed, any per-call logging can be disabled by running python in optimized mode. See the python documentation on the -O option and PYTHONOPTIMIZE environment variable.

You can benchmark the overhead yourself using the cpy2py_benchmark tools.

pypy2

15x15k

30x5k

300x1

pypy2

187 ± 1.5 us

228 ± 2.5 us

505 ± 51.8 us

pypy3

165 ± 1.3 us

209 ± 2.4 us

402 ± 8.0 us

python2.7

178 ± 0.6 us

139 ± 0.3 us

239 ± 7.6 us

python3.4

149 ± 0.4 us

118 ± 0.2 us

258 ± 8.0 us

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cpy2py-0.17.1.tar.gz (38.9 kB view hashes)

Uploaded Source

Built Distribution

cpy2py-0.17.1-py2.py3-none-any.whl (72.9 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page