Go-like features for Python and Cython
Project description
Package golang provides Go-like features for Python:
gpython is Python interpreter with support for lightweight threads.
go spawns lightweight thread.
chan and select provide channels with Go semantic.
func allows to define methods separate from class.
defer allows to schedule a cleanup from the main control flow.
error and package errors provide error chaining.
b and u provide way to make sure an object is either bytes or unicode.
gimport allows to import python modules by full path in a Go workspace.
Package golang.pyx provides similar features for Cython/nogil.
Additional packages and utilities are also provided to close other gaps between Python/Cython and Go environments.
GPython
Command gpython provides Python interpreter that supports lightweight threads via tight integration with gevent. The standard library of GPython is API compatible with Python standard library, but inplace of OS threads lightweight coroutines are provided, and IO is internally organized via libuv/libev-based IO scheduler. Consequently programs can spawn lots of coroutines cheaply, and modules like time, socket, ssl, subprocess etc - all could be used from all coroutines simultaneously, and in the same blocking way as if every coroutine was a full OS thread. This gives ability to scale programs without changing concurrency model and existing code.
Additionally GPython sets UTF-8 to be default encoding always, and puts go, chan, select etc into builtin namespace.
Goroutines and channels
go spawns a coroutine, or thread if gevent was not activated. It is possible to exchange data in between either threads or coroutines via channels. chan creates a new channel with Go semantic - either synchronous or buffered. Use chan.recv, chan.send and chan.close for communication. nilchan stands for nil channel. select can be used to multiplex on several channels. For example:
ch1 = chan() # synchronous channel
ch2 = chan(3) # channel with buffer of size 3
def _():
ch1.send('a')
ch2.send('b')
go(_)
ch1.recv() # will give 'a'
ch2.recv_() # will give ('b', True)
ch2 = nilchan # rebind ch2 to nil channel
_, _rx = select(
ch1.recv, # 0
ch1.recv_, # 1
(ch1.send, obj), # 2
ch2.recv, # 3
default, # 4
)
if _ == 0:
# _rx is what was received from ch1
...
if _ == 1:
# _rx is (rx, ok) of what was received from ch1
...
if _ == 2:
# we know obj was sent to ch1
...
if _ == 3:
# this case will be never selected because
# send/recv on nil channel block forever.
...
if _ == 4:
# default case
...
By default chan creates new channel that can carry arbitrary Python objects. However type of channel elements can be specified via chan(dtype=X) - for example chan(dtype=’C.int’) creates new channel whose elements are C integers. chan.nil(X) creates typed nil channel. Cython/nogil API explains how channels with non-Python dtypes, besides in-Python usage, can be additionally used for interaction in between Python and nogil worlds.
Methods
func decorator allows to define methods separate from class.
For example:
@func(MyClass)
def my_method(self, ...):
...
will define MyClass.my_method().
func can be also used on just functions, for example:
@func
def my_function(...):
...
Defer / recover / panic
defer allows to schedule a cleanup to be executed when current function returns. It is similar to try/finally but does not force the cleanup part to be far away in the end. For example:
wc = wcfs.join(zurl) │ wc = wcfs.join(zurl)
defer(wc.close) │ try:
│ ...
... │ ...
... │ ...
... │ finally:
│ wc.close()
If deferred cleanup fails, previously unhandled exception, if any, won’t be lost - it will be chained with (PEP 3134) and included into traceback dump even on Python2.
For completeness there is recover and panic that allow to program with Go-style error handling, for example:
def _():
r = recover()
if r is not None:
print("recovered. error was: %s" % (r,))
defer(_)
...
panic("aaa")
But recover and panic are probably of less utility since they can be practically natively modelled with try/except.
If defer is used, the function that uses it must be wrapped with @func decorator.
Errors
In concurrent systems operational stack generally differs from execution code flow, which makes code stack traces significantly less useful to understand an error. Pygolang provides support for error chaining that gives ability to build operational error stack and to inspect resulting errors:
error is error type that can be used by itself or subclassed. By providing .Unwrap() method, an error can optionally wrap another error this way forming an error chain. errors.Is reports whether an item in error chain matches target. fmt.Errorf provides handy way to build wrapping errors. For example:
e1 = error("problem")
e2 = fmt.Errorf("doing something for %s: %w", "joe", e1)
print(e2) # prints "doing something for joe: problem"
errors.Is(e2, e1) # gives True
# OpError is example class to represents an error of operation op(path).
class OpError(error):
def __init__(e, op, path, err):
e.op = op
e.path = path
e.err = err
# .Error() should be used to define what error's string is.
# it is automatically used by error to also provide both .__str__ and .__repr__.
def Error(e):
return "%s %s: %s" % (e.op, e.path, e.err)
# provided .Unwrap() indicates that this error is chained.
def Unwrap(e):
return e.err
mye = OpError("read", "file.txt", io.ErrUnexpectedEOF)
print(mye) # prints "read file.txt: unexpected EOF"
errors.Is(mye, io.EOF) # gives False
errors.Is(mye. io.ErrUnexpectedEOF) # gives True
Both wrapped and wrapping error can be of arbitrary Python type - not necessarily of error or its subclass.
error is also used to represent at Python level an error returned by Cython/nogil call (see Cython/nogil API) and preserves Cython/nogil error chain for inspection at Python level.
Pygolang error chaining integrates with Python error chaining and takes .__cause__ attribute into account for exception created via raise X from Y (PEP 3134).
Strings
b and u provide way to make sure an object is either bytes or unicode. b(obj) converts str/unicode/bytes obj to UTF-8 encoded bytestring, while u(obj) converts str/unicode/bytes obj to unicode string. For example:
b("привет мир") # -> gives bytes corresponding to UTF-8 encoding of "привет мир".
def f(s):
s = u(s) # make sure s is unicode, decoding as UTF-8(*) if it was bytes.
... # (*) but see below about lack of decode errors.
The conversion in both encoding and decoding never fails and never looses information: b(u(·)) and u(b(·)) are always identity for bytes and unicode correspondingly, even if bytes input is not valid UTF-8.
Import
gimport provides way to import python modules by full path in a Go workspace.
For example
lonet = gimport('lab.nexedi.com/kirr/go123/xnet/lonet')
will import either
lab.nexedi.com/kirr/go123/xnet/lonet.py, or
lab.nexedi.com/kirr/go123/xnet/lonet/__init__.py
located in src/ under $GOPATH.
Cython/nogil API
Cython package golang provides nogil API with goroutines, channels and other features that mirror corresponding Python package. Cython API is not only faster compared to Python version, but also, due to nogil property, allows to build concurrent systems without limitations imposed by Python’s GIL. All that while still programming in Python-like language. Brief description of Cython/nogil API follows:
go spawns new task - a coroutine, or thread, depending on activated runtime. chan[T] represents a channel with Go semantic and elements of type T. Use makechan[T] to create new channel, and chan[T].recv, chan[T].send, chan[T].close for communication. nil stands for nil channel. select can be used to multiplex on several channels. For example:
cdef nogil:
struct Point:
int x
int y
void worker(chan[int] chi, chan[Point] chp):
chi.send(1)
cdef Point p
p.x = 3
p.y = 4
chp.send(p)
void myfunc():
cdef chan[int] chi = makechan[int]() # synchronous channel of integers
cdef chan[Point] chp = makechan[Point](3) # channel with buffer of size 3 and Point elements
go(worker, chi, chp)
i = chi.recv() # will give 1
p = chp.recv() # will give Point(3,4)
chp = nil # rebind chp to nil channel
cdef cbool ok
cdef int j = 33
_ = select([
chi.recvs(&i), # 0
chi.recvs(&i, &ok), # 1
chi.sends(&j), # 2
chp.recvs(&p), # 3
default, # 4
])
if _ == 0:
# i is what was received from chi
...
if _ == 1:
# (i, ok) is what was received from chi
...
if _ == 2:
# we know j was sent to chi
...
if _ == 3:
# this case will be never selected because
# send/recv on nil channel block forever.
...
if _ == 4:
# default case
...
Python channels are represented by pychan cdef class. Python channels that carry non-Python elements (pychan.dtype != DTYPE_PYOBJECT) can be converted to Cython/nogil chan[T] via pychan.chan_*(). Similarly Cython/nogil chan[T] can be wrapped into pychan via pychan.from_chan_*(). This provides interaction mechanism in between nogil and Python worlds. For example:
def myfunc(pychan pych):
if pych.dtype != DTYPE_INT:
raise TypeError("expected chan[int]")
cdef chan[int] ch = pych.chan_int() # pychan -> chan[int]
with nogil:
# use ch in nogil code. Both Python and nogil parts can
# send/receive on the channel simultaneously.
...
def mytick(): # -> pychan
cdef chan[int] ch
with nogil:
# create a channel that is connected to some nogil task of the program
ch = ...
# wrap the channel into pychan. Both Python and nogil parts can
# send/receive on the channel simultaneously.
cdef pychan pych = pychan.from_chan_int(ch) # pychan <- chan[int]
return pych
error is the interface that represents errors. errors.New and fmt.errorf provide way to build errors from text. An error can optionally wrap another error by implementing errorWrapper interface and providing .Unwrap() method. errors.Is reports whether an item in error chain matches target. fmt.errorf with %w specifier provide handy way to build wrapping errors. For example:
e1 = errors.New("problem")
e2 = fmt.errorf("doing something for %s: %w", "joe", e1)
e2.Error() # gives "doing something for joe: problem"
errors.Is(e2, e1) # gives True
An error can be exposed to Python via pyerror cdef class wrapper instantiated by pyerror.from_error(). pyerror preserves Cython/nogil error chain for inspection by Python-level error.Is.
panic stops normal execution of current goroutine by throwing a C-level exception. On Python/C boundaries C-level exceptions have to be converted to Python-level exceptions with topyexc. For example:
cdef void _do_something() nogil:
...
panic("bug") # hit a bug
# do_something is called by Python code - it is thus on Python/C boundary
cdef void do_something() nogil except +topyexc:
_do_something()
def pydo_something():
with nogil:
do_something()
See libgolang.h and golang.pxd for details of the API. See also testprog/golang_pyx_user/ for demo project that uses Pygolang in Cython/nogil mode.
Additional packages and utilities
The following additional packages and utilities are also provided to close gaps between Python/Cython and Go environments:
Concurrency
In addition to go and channels, the following packages are provided to help handle concurrency in structured ways:
golang.context (py, pyx) provides contexts to propagate deadlines, cancellation and task-scoped values among spawned goroutines [*].
golang.sync (py, pyx) provides sync.WorkGroup to spawn group of goroutines working on a common task. It also provides low-level primitives - for example sync.Once, sync.WaitGroup, sync.Mutex and sync.RWMutex - that are sometimes useful too.
golang.time (py, pyx) provides timers integrated with channels.
golang.os.signal (py, pyx) provides signal handling via channels.
String conversion
qq (import from golang.gcompat) provides %q functionality that quotes as Go would do. For example the following code will print name quoted in “ without escaping printable UTF-8 characters:
print('hello %s' % qq(name))
qq accepts both str and bytes (unicode and str on Python2) and also any other type that can be converted to str.
Package golang.strconv provides direct access to conversion routines, for example strconv.quote and strconv.unquote.
Benchmarking and testing
py.bench allows to benchmark python code similarly to go test -bench and py.test. For example, running py.bench on the following code:
def bench_add(b):
x, y = 1, 2
for i in xrange(b.N):
x + y
gives something like:
$ py.bench --count=3 x.py ... pymod: bench_add.py Benchmarkadd 50000000 0.020 µs/op Benchmarkadd 50000000 0.020 µs/op Benchmarkadd 50000000 0.020 µs/op
Package golang.testing provides corresponding runtime bits, e.g. testing.B.
py.bench produces output in Go benchmark format, and so benchmark results can be analyzed and compared with standard Go tools, for example with benchstat. Additionally package golang.x.perf.benchlib can be used to load and process such benchmarking data in Python.
GPython options
GPython mimics and supports most of Python command-line options, like gpython -c <commands> to run Python statements from command line, or gpython -m <module> to execute a module. Such options have the same meaning as in standard Python and are not documented here.
GPython-specific options and environment variables are listed below:
- -X gpython.runtime=(gevent|threads)
Specify which runtime GPython should use. gevent provides lightweight coroutines, while with threads go spawns full OS thread. gevent is default. The runtime to use can be also specified via $GPYTHON_RUNTIME environment variable.
Pygolang change history
0.1 (2022-01-26)
Add os.signal package that provides signal handing via nogil channels. This allows to build concurrent systems without limitation of Python’s standard signal module: signal delivery is not delayed, potentially indefinitely, if main thread is blocked or is busy doing any non-Python work (commit 1, 2, 3, example usage).
Fix segmentation-fault crashes on unhandled panic with gevent backend (commit, greenlet bug).
Fix print(qq(·)) crash on Python2 (commit).
0.0.9 (2021-12-08)
Fix deadlock when new context is created from already-canceled parent (commit 1, 2).
Add support for “with” statement in sync.WorkGroup. This is sometimes handy and is referred to as “structured concurrency” in Python world (commit, discussion).
Fix strconv.unqoute to handle all input that Go strconv.Qoute might produce (commit).
More fixes for gpython to be compatible with CPython in how it handles program on stdin, interactive session and __main__ module setup (commit 1, 2, 3, 4, 5).
0.0.8 (2020-12-02)
0.0.7 (2020-09-22)
Add way to run gpython with either gevent or threads runtime. This allows gpython usage without forcing projects to switch from threads to greenlets (commit 1, 2, 3).
Fix gpython to be more compatible with CPython on command line handling (commit 1, 2, 3, 4, 5, 6, 7).
Teach qq to be usable with both bytes and str format whatever type qq’s argument is (commit 1, 2).
Teach recover to always return exception with .__traceback__ set even on Python2 (commit).
Fix pyx.build for develop install (commit).
Fix pyx.build on macOS (commit).
Add tests for IPython and Pytest integration patches (commit 1, 2, 3, 4, 5, 6, 7).
Fix ThreadSanitizer/AddressSanitizer support on upcoming Debian 11 (commit).
0.0.6 (2020-02-28)
Provide support for error chaining. In concurrent systems operational stack generally differs from execution code flow, which makes code stack traces significantly less useful to understand an error. Error chaining gives ability to build operational error stack and to inspect resulting errors. (commit 1, 2, 3, 4, 5, 6, overview 1, overview 2).
Provide unicode ↔ bytes conversion: b(obj) converts str/unicode/bytes obj to UTF-8 encoded bytestring, while u(obj) converts str/unicode/bytes obj to unicode string. The conversion in both encoding and decoding never fails and never looses information: b(u(·)) and u(b(·)) are always identity for bytes and unicode correspondingly, even if bytes input is not valid UTF-8. (commit 1, 2, 3, 4, 5, 6, 7).
Provide nil as alias for nullptr and NULL (commit 1, 2, 3, 4).
Add io package with io.EOF and io.ErrUnexpectedEOF (commit).
Correct cxx.dict API to follow libgolang comma-ok style (commit).
Provide pyx.build.DSO for projects to build dynamic libraries that use/link-to libgolang (commit 1, 2).
Fix pyx.build.build_ext to allow customization (commit).
0.0.5 (2019-11-27)
Add support for typed Python channels. For example chan(dtype=’C.int’) creates channel whose elements type is C int instead of Python object. Besides providing runtime type-safety, this allows to build interaction in between Python and nogil worlds (commit 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11).
Provide automatic memory management for C++/Cython/nogil classes. Used approach complements “Automatic multithreaded-safe memory managed classes in Cython” (Gwenaël Samain et al. 2019, blog post) (commit 1, 2, 3, 4, 5, 6, 7).
Provide minimal support for interfaces with empty and error interfaces provided by base library (commit 1, 2).
Provide sync.Mutex and sync.Sema as part of both Python and Cython/nogil API (commit 1, 2, 3, 4, 5, 6).
Provide C++/Cython/nogil API for time package. Python-level time becomes a small wrapper around Cython/nogil one (commit 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14).
Provide C++/Cython/nogil API for context package. Python-level context becomes a small wrapper around Cython/nogil one (commit 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15).
Provide C++/Cython/nogil API for sync package. Python-level sync becomes a small wrapper around Cython/nogil one (commit 1, 2, 3, 4, 5, 6, 7, 8, 9).
Add errors package with errors.New to create new error with provided text (commit).
Add fmt package with fmt.sprintf and fmt.errorf to format text into strings and errors (commit).
Add strings package with utilities like strings.has_prefix, strings.split and similar (commit).
Add cxx package with cxx.dict and cxx.set providing ergonomic interface over STL hash map and set (commit).
Teach defer to chain exceptions (PEP 3134) and adjust traceback dumps to include exception cause/context even on Python2 (commit 1, 2, 3, 4, 5).
Provide build_ext as part of pyx.build package API. This allows projects to customize the way their Pygolang-based extensions are built (commit 1, 2).
Fix select to not leak object reference on error path (commit).
Fix gevent runtime to preserve Python exception state during runtime calls (commit 1, 2).
0.0.4 (2019-09-17)
Add ThreadSanitizer, AddressSanitizer and Python debug builds to testing coverage (commit).
Fix race bugs in close, recv and select (commit 1, 2, 3, 4, 5, 6). A 25-years old race condition in Python was also discovered while doing quality assurance on concurrency (commit 7, Python bug, PyPy bug).
If C-level panic causes termination, its argument is now printed (commit).
0.0.3 (2019-08-29)
Provide Cython/nogil API with goroutines and channels. Cython API is not only faster compared to Python version, but also, due to nogil property, allows to build concurrent systems without limitations imposed by Python’s GIL. This work was motivated by wendelin.core v2, which, due to its design, would deadlock if it tries to take the GIL in its pinner thread. Implementation of Python-level goroutines and channels becomes tiny wrapper around Cython/nogil API. This brings in ~5x speedup to Python-level golang package along the way (commit 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27).
Provide way to install Pygolang with extra requirements in the form of pygolang[<package>]. For example pygolang[x.perf.benchlib] additionally selects NumPy, pygolang[pyx.build] - everything needed by build system, and pygolang[all] selects everything (commit).
Improve tests to exercise the implementation more thoroughly in many places (commit 1, 2, 3, 4, 5, 6).
Fix race bugs in buffered channel send and receive (commit 1, 2).
Fix deadlock in sync.WorkGroup tests (commit).
Fix @func(cls) def name not to override name in calling context (commit).
Fix sync.WorkGroup to propagate all exception types, not only those derived from Exception (commit).
Replace threading.Event with chan in sync.WorkGroup implementation. This removes reliance on outside semaphore+waitlist code and speeds up sync.WorkGroup along the way (commit).
Speedup sync.WorkGroup by not using @func at runtime (commit).
Add benchmarks for chan, select, @func and defer (commit).
This release is dedicated to the memory of Вера Павловна Супрун.
0.0.2 (2019-05-16)
0.0.1 (2019-05-09)
0.0.0.dev8 (2019-03-24)
0.0.0.dev7 (2019-01-16)
Provide gpython interpreter, that sets UTF-8 as default encoding, integrates gevent and puts go, chan, select etc into builtin namespace (commit).
0.0.0.dev6 (2018-12-13)
0.0.0.dev5 (2018-10-30)
Fix select bug that was causing several cases to be potentially executed at the same time (commit 1, 2, 3).
Add defer and recover (commit). The implementation is partly inspired by work of Denis Kolodin (1, 2).
Fix @method on Python3 (commit).
A leaked goroutine no longer prevents whole program to exit (commit 1, 2).
0.0.0.dev4 (2018-07-04)
Add py.bench program and golang.testing package with corresponding bits (commit).
0.0.0.dev3 (2018-07-02)
0.0.0.dev2 (2018-06-20)
Turn into full pygolang: go, chan, select, method and gcompat.qq are provided in addition to gimport (commit). The implementation is not very fast, but should be working correctly including select - select sends for synchronous channels.
0.0.0.dev1 (2018-05-21)
Initial release; gimport functionality only (commit).
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
