human-oriented ast parser/unparser
Human-oriented abstract syntax tree (AST) parser/unparser for Python 3 that doesn’t discard comments.
This package provides new AST node types (Comment and Directive) which inherit from nodes in typed_ast.ast3 module. Additionally, it provides implementation of parser and unparser for the extended AST allowing straightforward readable code generation.
Simple example of how to use this package:
from horast import parse, unparse tree = parse("""a = 1 # a equals one after this""") print(unparse(tree)) # this will print the code with original comment
More examples in examples.ipynb.
Parser is based on built-in tokenize module, as well as community packages asttokens and typed_ast.
Unparser is essentially an extension of Unparser class from static_typing package.
Nodes provided and handled by horast are listed below.
Not implemented yet.
This node type is intended to store consecutive full-line comments in a single AST node, therefore simplifying handling of large blocks of comments.
Since Python doesn’t offer a direct way to convey directives in code, comments beginning with any of the following prefixes will not be classified as typical comments, but as directives:
However in Python code they still remain as usual comments.
For example, the comments in the following code will all be classified as directives.
#ifdef DEBUG debugging = True #else debugging = False #fi
However, when executed as Python, debugging will always end up False because directives are preserved as usual comments in Python and therefore they are ignored.
Therefore, the Directive node is not meant to enable preprocessing of Python, at least for now.
Note: the prefix is checked exactly. See the following example:
#if something # if some other thing
The comment in the first line will become Directive object, while the one on the second like will become Comment object.
Currently, this node is meant to work towards AST compatibility between Python and other languages, to aide code generation from Python AST into code in other languages – one such use case is *transpyle* project.
Pragma nodes follow the same general rules as Directive nodes, i.e. they are stored in Python code as comments, but a comment will be classified as a pragma when it’s prefixed with a predefined prefix:
- " pragma:"
Additionally, two subclasses of Pragma are defined in horast, each with its own prefix that builds upon the generic pragma prefix:
- OpenMpPragma class defines prefix " pragma: omp" and stores OpenMP pragmas.
- OpenAccPragma class defines prefix " pragma: acc" and stores OpenACC pragmas.
A code snippet below contains all 3 pragma types.
# pragma: once use_openmp = True use_openacc = True ... a, b = np.ndarray(...) c = np.zeros(...) # pragma: acc parallel copyin(a,b) copyout(c) # pragma: acc loop gang for y in range(ymax): # type: np.int32 # pragma: acc loop worker for i in range(imax): # type: np.int32 # pragma: acc loop vector reduction(+: c[y][x]) for x in range(xmax): # type: np.int32 c[y, x] += a[y, i] * b[i, x] # pragma: acc end parallel ... # pragma: omp parallel do for i in range(input_data.size): # type: int # here we compute spam spam spam heavy_compute(input_data[i]) ...
And thus, in the example above:
- all comments starting with " pragma: omp" become OpenMpPragma objects,
- all comments starting with " pragma: acc" become OpenAccPragma objects,
- all other comments starting with " pragma:" become Pragma objects,
- type comments are ignored, and
- all other comments become Comment objects.
Additionally, horast module provides an extensible infrastructure to define custom Pragma subclasses, enabling user to define their own pragmas for experimentation. The provided OpenMP and OpenACC pragma definitions serve as examples of how to use this feature.
Similarly to how pragmas are handled, if a comment begins with " include:" prefix, it will be classified as a special kind of include directive.
Again, this will be preserved as comment in Python code, but it’s useful for enhancing syntactic compatibility between Python and other, especially statically compiled languages.
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size horast-0.4.0-py3-none-any.whl (19.9 kB)||File type Wheel||Python version py3||Upload date||Hashes View|
|Filename, size horast-0.4.0.tar.gz (29.3 kB)||File type Source||Python version None||Upload date||Hashes View|