=================================================== Pygolang - Go-like features for Python and Cython

Package golang provides Go-like features for Python:

gpython is Python interpreter with support for lightweight threads.
go spawns lightweight thread.
chan and select provide channels with Go semantic.
func allows to define methods separate from class.
defer allows to schedule a cleanup from the main control flow.
error and package errors provide error chaining.
b, u and bstr/ustr provide uniform UTF8-based approach to strings.
gimport allows to import python modules by full path in a Go workspace.

Package golang.pyx provides__ similar features for Cython/nogil.

__ Cython/nogil API_

Additional packages and utilities are also provided__ to close other gaps between Python/Cython and Go environments.

__ Additional packages and utilities_

.. contents:: :depth: 1

GPython

Command gpython provides Python interpreter that supports lightweight threads via tight integration with gevent__. The standard library of GPython is API compatible with Python standard library, but inplace of OS threads lightweight coroutines are provided, and IO is internally organized via libuv__/libev__-based IO scheduler. Consequently programs can spawn lots of coroutines cheaply, and modules like time, socket, ssl, subprocess etc - all could be used from all coroutines simultaneously, and in the same blocking way as if every coroutine was a full OS thread. This gives ability to scale programs without changing concurrency model and existing code.

__ http://www.gevent.org/ __ http://libuv.org/ __ http://software.schmorp.de/pkg/libev.html

Additionally GPython sets UTF-8 to be default encoding always, and puts go, chan, select etc into builtin namespace.

.. note::

GPython is optional and the rest of Pygolang can be used from under standard Python too. However without gevent integration go spawns full - not lightweight - OS thread. GPython can be also used with threads - not gevent - runtime. Please see GPython options_ for details.

Goroutines and channels

go spawns a coroutine, or thread if gevent was not activated. It is possible to exchange data in between either threads or coroutines via channels. chan creates a new channel with Go semantic - either synchronous or buffered. Use chan.recv, chan.send and chan.close for communication. nilchan stands for nil channel. select can be used to multiplex on several channels. For example::

ch1 = chan()    # synchronous channel
ch2 = chan(3)   # channel with buffer of size 3

def _():
    ch1.send('a')
    ch2.send('b')
go(_)

ch1.recv()      # will give 'a'
ch2.recv_()     # will give ('b', True)

ch2 = nilchan   # rebind ch2 to nil channel
_, _rx = select(
    ch1.recv,           # 0
    ch1.recv_,          # 1
    (ch1.send, obj),    # 2
    ch2.recv,           # 3
    default,            # 4
)
if _ == 0:
    # _rx is what was received from ch1
    ...
if _ == 1:
    # _rx is (rx, ok) of what was received from ch1
    ...
if _ == 2:
    # we know obj was sent to ch1
    ...
if _ == 3:
    # this case will be never selected because
    # send/recv on nil channel block forever.
    ...
if _ == 4:
    # default case
    ...

By default chan creates new channel that can carry arbitrary Python objects. However type of channel elements can be specified via chan(dtype=X) - for example chan(dtype='C.int') creates new channel whose elements are C integers. chan.nil(X) creates typed nil channel. Cython/nogil API_ explains how channels with non-Python dtypes, besides in-Python usage, can be additionally used for interaction in between Python and nogil worlds.

Methods

func decorator allows to define methods separate from class.

For example::

@func(MyClass) def my_method(self, ...): ...

will define MyClass.my_method().

func can be also used on just functions, for example::

@func def my_function(...): ...

Defer / recover / panic

defer allows to schedule a cleanup to be executed when current function returns. It is similar to try/finally but does not force the cleanup part to be far away in the end. For example::

wc = wcfs.join(zurl) │ wc = wcfs.join(zurl) defer(wc.close) │ try: │ ... ... │ ... ... │ ... ... │ finally: │ wc.close()

If deferred cleanup fails, previously unhandled exception, if any, won't be lost - it will be chained with (PEP 3134__) and included into traceback dump even on Python2.

__ https://www.python.org/dev/peps/pep-3134/

For completeness there is recover and panic that allow to program with Go-style error handling, for example::

def (): r = recover() if r is not None: print("recovered. error was: %s" % (r,)) defer()

...

panic("aaa")

But recover and panic are probably of less utility since they can be practically natively modelled with try/except.

If defer is used, the function that uses it must be wrapped with @func decorator.

Errors

In concurrent systems operational stack generally differs from execution code flow, which makes code stack traces significantly less useful to understand an error. Pygolang provides support for error chaining that gives ability to build operational error stack and to inspect resulting errors:

error is error type that can be used by itself or subclassed. By providing .Unwrap() method, an error can optionally wrap another error this way forming an error chain. errors.Is reports whether an item in error chain matches target. fmt.Errorf provides handy way to build wrapping errors. For example::

e1 = error("problem") e2 = fmt.Errorf("doing something for %s: %w", "joe", e1) print(e2) # prints "doing something for joe: problem" errors.Is(e2, e1) # gives True

OpError is example class to represents an error of operation op(path).

class OpError(error): def init(e, op, path, err): e.op = op e.path = path e.err = err

  # .Error() should be used to define what error's string is.
  # it is automatically used by error to also provide both .__str__ and .__repr__.
  def Error(e):
     return "%s %s: %s" % (e.op, e.path, e.err)

  # provided .Unwrap() indicates that this error is chained.
  def Unwrap(e):
     return e.err

mye = OpError("read", "file.txt", io.ErrUnexpectedEOF) print(mye) # prints "read file.txt: unexpected EOF" errors.Is(mye, io.EOF) # gives False errors.Is(mye. io.ErrUnexpectedEOF) # gives True

Both wrapped and wrapping error can be of arbitrary Python type - not necessarily of error or its subclass.

error is also used to represent at Python level an error returned by Cython/nogil call (see Cython/nogil API_) and preserves Cython/nogil error chain for inspection at Python level.

Pygolang error chaining integrates with Python error chaining and takes .__cause__ attribute into account for exception created via raise X from Y (PEP 3134__).

__ https://www.python.org/dev/peps/pep-3134/

Strings

Pygolang, similarly to Go, provides uniform UTF8-based approach to strings with the idea to make working with byte- and unicode- strings easy and transparently interoperable:

bstr is byte-string: it is based on bytes and can automatically convert to/from unicode [*]_.
ustr is unicode-string: it is based on unicode and can automatically convert to/from bytes.

The conversion, in both encoding and decoding, never fails and never looses information: bstr→ustr→bstr and ustr→bstr→ustr are always identity even if bytes data is not valid UTF-8.

Both bstr and ustr represent stings. They are two different representations of the same entity.

Semantically bstr is array of bytes, while ustr is array of unicode-characters. Accessing their elements by [index] and iterating them yield byte and unicode character correspondingly [*]_. However it is possible to yield unicode character when iterating bstr via uiter, and to yield byte character when iterating ustr via biter. In practice bstr + uiter is enough 99% of the time, and ustr only needs to be used for random access to string characters. See Strings, bytes, runes and characters in Go__ for overview of this approach.

__ https://blog.golang.org/strings

Operations in between bstr and ustr/unicode / bytes/bytearray coerce to bstr, while operations in between ustr and bstr/bytes/bytearray / unicode coerce to ustr. When the coercion happens, bytes and bytearray, similarly to bstr, are also treated as UTF8-encoded strings.

bstr and ustr are meant to be drop-in replacements for standard str/unicode classes. They support all methods of str/unicode and in particular their constructors accept arbitrary objects and either convert or stringify them. For cases when no stringification is desired, and one only wants to convert bstr/ustr / unicode/bytes/bytearray, or an object with buffer interface [*]_, to Pygolang string, b and u provide way to make sure an object is either bstr or ustr correspondingly.

Usage example::

s = b('привет') # s is bstr corresponding to UTF-8 encoding of 'привет'. s += ' мир' # s is b('привет мир') for c in uiter(s): # c will iterate through ... # [u(_) for _ in ('п','р','и','в','е','т',' ','м','и','р')]

Pygolang

Install / Use

README