horast 0.4.0

human-oriented ast parser/unparser

Human-oriented abstract syntax tree (AST) parser/unparser for Python 3 that doesn’t discard comments.

This package provides new AST node types (Comment and Directive) which inherit from nodes in typed_ast.ast3 module. Additionally, it provides implementation of parser and unparser for the extended AST allowing straightforward readable code generation.

Simple example of how to use this package:

from horast import parse, unparse

tree = parse("""a = 1  # a equals one after this""")
print(unparse(tree))
# this will print the code with original comment

More examples in examples.ipynb.

technical details

Parser is based on built-in tokenize module, as well as community packages asttokens and typed_ast.

Unparser is essentially an extension of Unparser class from static_typing package.

Nodes provided and handled by horast are listed below.

Comment

Full line as well as end-of-line comments are parsed/unparsed correctly when they are outside of multi-line expressions.

Currently, handling of comments within multi-line expressions is implemented only partially.

BlockComment

Not implemented yet.

This node type is intended to store consecutive full-line comments in a single AST node, therefore simplifying handling of large blocks of comments.

Directive

Since Python doesn’t offer a direct way to convey directives in code, comments beginning with any of the following prefixes will not be classified as typical comments, but as directives:

• if

• else

• endif

• def

• undef

• ifdef

• ifndef

However in Python code they still remain as usual comments.

For example, the comments in the following code will all be classified as directives.

#ifdef DEBUG
debugging = True
#else
debugging = False
#fi

However, when executed as Python, debugging will always end up False because directives are preserved as usual comments in Python and therefore they are ignored.

Therefore, the Directive node is not meant to enable preprocessing of Python, at least for now.

Note: the prefix is checked exactly. See the following example:

#if something
# if some other thing

The comment in the first line will become Directive object, while the one on the second like will become Comment object.

Currently, this node is meant to work towards AST compatibility between Python and other languages, to aide code generation from Python AST into code in other languages – one such use case is *transpyle* project.

Pragma

Pragma nodes follow the same general rules as Directive nodes, i.e. they are stored in Python code as comments, but a comment will be classified as a pragma when it’s prefixed with a predefined prefix:

• " pragma:"

Additionally, two subclasses of Pragma are defined in horast, each with its own prefix that builds upon the generic pragma prefix:

• OpenMpPragma class defines prefix " pragma: omp" and stores OpenMP pragmas.

• OpenAccPragma class defines prefix " pragma: acc" and stores OpenACC pragmas.

A code snippet below contains all 3 pragma types.

# pragma: once
use_openmp = True
use_openacc = True
...
a, b = np.ndarray(...)
c = np.zeros(...)
# pragma: acc parallel copyin(a,b) copyout(c)
# pragma: acc loop gang
for y in range(ymax): # type: np.int32
    # pragma: acc loop worker
    for i in range(imax): # type: np.int32
        # pragma: acc loop vector reduction(+: c[y][x])
        for x in range(xmax): # type: np.int32
            c[y, x] += a[y, i] * b[i, x]
# pragma: acc end parallel
...
# pragma: omp parallel do
for i in range(input_data.size):  # type: int
    # here we compute spam spam spam
    heavy_compute(input_data[i])
...

And thus, in the example above:

• all comments starting with " pragma: omp" become OpenMpPragma objects,

• all comments starting with " pragma: acc" become OpenAccPragma objects,

• all other comments starting with " pragma:" become Pragma objects,

• type comments are ignored, and

• all other comments become Comment objects.

Additionally, horast module provides an extensible infrastructure to define custom Pragma subclasses, enabling user to define their own pragmas for experimentation. The provided OpenMP and OpenACC pragma definitions serve as examples of how to use this feature.

Include

Similarly to how pragmas are handled, if a comment begins with " include:" prefix, it will be classified as a special kind of include directive.

Again, this will be preserved as comment in Python code, but it’s useful for enhancing syntactic compatibility between Python and other, especially statically compiled languages.

requirements

CPython 3.5 or later.

Python libraries as specified in requirements.txt.

Building and running tests additionally requires packages listed in test_requirements.txt.