Skip to main content

Python code-generation for Python

Project description

GitHub Workflow Status GitHub Workflow Status PyPI PyPI - Downloads PyPI - Python Version Coveralls License Badge Count

This is a Python code-generation module.

  • Generates any Python statement/expression
  • Places parens to ensure expression priorities are unchanged
  • Places extra newlines before/after class/function definitions to conform with PEP 8
  • 100% coverage of type hints, passing strict Pyright checks on every PR
  • Meaningful type hierarchy inspired by Python grammar to guarantee syntax validity almost in every case.
  • Covered with diamonds tests completely
  • Compatible with wordstreamer renderables via a wrapper

Installation

Just install gekkota package, e.g. with python -m pip install gekkota (or any other package manager of your choice)

Rendering and configuration

To render any Renderable into a string, you could use a few approaches:

  • str(renderable): renders a Renderable with default configuration (check below)
  • renderable.render_str(): also default configuration
  • renderable.render_str(config): overrides default config options with provided in config mapping. Unspecified keys remain at default values

Here is current default config:

default_config: Config = {
    "tab_size": 4,  # how much chars to use in indentation
    "compact": False,  # if True, renders without redundant whitespace (e.g "for [i, e] in enumerate(a)" renders as "for[i,e]in enumerate(a)")
    "tab_char": " ",  # character used for indentation
    # "place_semicolons" and "inline_small_stmts" options have some performance impact, since those require checking for newlines in token stream before re-streaming tokens.
    # this impact is probably negligible, but be aware of it
    "place_semicolons": False,  # if True, semicolons are placed after one-line statements
    "inline_small_stmts": False,  # if True, one-line statements are inlined. Overrides "place_semicolons" if True.
}

Expressions

Basic expressions

Your starting points would be to_expression and Name:

from gekkota import Name, to_expression

# Name(name: str, annotation: Optional[Expression] = None)
# to_expression(value: int | float | complex | str | bytes | bool | None)

a = Name("a")
b = Name("b")
six = to_expression(6)

# prints 'a + b * 6'
print(
    (a + b * six)
)

Name, as many other classes in the module, is an Expression instance

Expressions support most operations to combine with other expressions.
Exceptions are:

  • Attribute reference: for that you should use Expression.getattr(other: str)
  • Indexing: Expression.index(index: Expression)
  • Slicing: Expression.index(index: Union[SliceExpr, Sequence[SliceExpr]])
  • Equality / Inequality: Expression.eq(right_hand_side) and Expression.neq(right_hand_side) respectively
  • is: Expression.is_(right_hand_side),
  • is not: Expression.is_not(right_hand_side)
  • in: Expression.in_(right_hand_side)
  • not in: Expression.not_in(right_hand_side)
  • and: Expression.and_(right_hand_side)
  • or: Expression.or_(right_hand_side)
  • await: Expression.await_()
  • := assignment: Expression.assign(value)
  • Ternary operator: Expression.if_(condition, alternative)

For example:

from gekkota import Name

a = Name("a")
b = Name("b")

expression = a.await_().in_(b)

print(expression) # await a in b

For any other operation on expressions you can just use familiar Python syntax:

from gekkota import Name

a = Name("a")
b = Name("b")
c = Name("c")

print(
    (a + b * c / a(b, c)) # 'a + b * c / a(b, c)'
)

Sequences

Most convenient way to create sequence literals is, again, to_expression:

from gekkota import to_expression, Name

a = Name("a")
b = Name("b")

print(
    to_expression( (a, b, 6) ), # '(a, b, 6)' (notice that to_expression is recursive)
    to_expression( (a, ) ),     # '(a, )'
    to_expression([a, b]),      # '[a, b]'
    to_expression([]),          # '[]'
    to_expression({a: b}),      # '{a: b}'
    to_expression(set()),       # 'set()'
    to_expression([a, [a, b]]), # '[a, [a, b]]'
)

If you want to have more precise control, you can use TupleExpr, ListExpr, SetExpr and DictExpr for this. All have same constructor signature: (values: Sequence[Expression]) (except DictExpr, which has KeyValue values)

To create comprehensions:

from gekkota import Name, GeneratorFor, GeneratorIf
from gekkota import (
    ListComprehension,
    DictComprehension,
    KeyValue,
    SetComprehension, # same usage as ListComprehension
)

a, b, c, d = map(Name, "abcd")

# ListComprehension(generator_or_expr: GeneratorBase | Expression, parts: Sequence[GeneratorPart] = ())
print(
    ListComprehension(
        a,
        [
            # GeneratorFor(target: AssignmentTarget, iterator: Expression, *, is_async: bool = False)
            GeneratorFor(b, c),

            # GeneratorIf(condition: Expression)
            GeneratorIf(b.eq(d))
        ]
    )
) # [a for b in c if b == d]

# DictComprehension(generator_or_expr: GeneratorBase | KeyValue, parts: Sequence[GeneratorPart] = ())
# GeneratorPart == GeneratorFor | GeneratorIf
print(
    DictComprehension(KeyValue(a, b), [GeneratorFor(c, d), GeneratorIf(b.eq(d))])
) # {a: b for c in d if b == d}

Keyword call args

Use CallArg to provide keyword call args:

from gekkota import Name, to_expression

print_ = Name("print")

# CallArg(name: str, value: Optional[Expression] = None)
print(
    print_(
        Name("a"),
        CallArg("b"),
        CallArg("sep", to_expression(", "))
    )
) # print(a, b, sep=', ')

Type hints

To annotate a name, just pass an additional parameter to Name:

from gekkota import Name

a = Name("a", Name("int"))

print(a) # a: int

New in version 0.6: Name is now generic, depending on the presence of the annotation.
Some constructors now accept only annotated Name, while some others accept only unannotated (depends on syntax features allowed).
This allows to ensure type soundness without breaking changes in API.

Statements

To render program code (with multiple statements), use Code:

from gekkota import Code, Assignment, Name

a = Name("a")

six = Literal(6)

create_variable = Assignment(
    [Name("a")],
    six + six
)

print_variable = Name("print")(a)

print(
    Code([
        create_variable,
        print_variable,
    ])
)
# prints:
# a = 6 + 6
# print(a)

To render a block of code, use Block:

from gekkota import Block, IfStmt, Assignment, Name

a = Name("a")
b = Name("b")

six = Literal(6)

create_variable = Assignment(
    [Name("a")],
    six + six
)

print_variable = Name("print")(a)

print(
    IfStmt(
        b,
        Block([
            create_variable,
            print_variable,
        ])
    )
)
# prints:
# if b:
#     a = 6 + 6
#     print(a)

If the difference between two is not obvious: Code just renders statements on separate lines, while block also adds a newline before the first statement and indentation to every line. Moreover, Code([]) renders into "", while Block([]) — into "\n pass"

Small statements

Here is an example of a few small statements:

from gekkota import Name, SequenceExpr
from gekkota import (
    ReturnStmt,
    DelStmt,
    AssertStmt,
    BreakStmt,
    ContinueStmt,
    YieldStmt,
    YieldFromStmt,
    NonLocalStmt,
    GlobalStmt,
    PassStmt,
    RaiseStmt,
    AsyncStmt
)

a, b, c = map(Name, "abc")

print(ReturnStmt(a)) # 'return a'

print(YieldStmt(a)) # 'yield a'
print(YieldFromStmt(b)) # 'yield from b'

print(DelStmt(a, b)) # 'del a, b'

print(AssertStmt(a)) # 'assert a'

print(BreakStmt()) # 'break'

print(ContinueStmt()) # 'continue'

print(GlobalStmt(a, b)) # 'global a, b'
print(NonLocalStmt(a, b)) # 'nonlocal a, b'

print(PassStmt()) # 'pass'

print(RaiseStmt()) # 'raise'
print(RaiseStmt(a)) # 'raise a'
print(RaiseStmt(a, b)) # 'raise a from b'

Type statement

Gekkota 0.6 supports PEP 695 which adds new syntax, including type statement:

type Point = tuple[float, float]

To generate such statement in gekkota, use gekkota.annotations.TypeStmt:

from gekkota import TypeStmt

# type Point = tuple[float, float]
print(
    TypeStmt(
        "Point",
        [],
        TupleExpr([Name("float"), Name("float")])
    )
)

The second argument is a sequence of type parameters (for genric types): Sequence[TypeParam]

You can use several types as TypeParam:

from gekkota import TypeParam, TypeStmt, Name, TypeVarParam, TypeVarTupleParam, ParamSpecParam

def make_type(type_params: Sequence[TypeParam]):
    return TypeStmt(
        "CoolAlias",
        type_params,
        Name("int")
    )


print(make_type([])) # type CoolAlias = int

# (both) type CoolAlias[T] = int
print(make_type([Name("T")]))
print(
    make_type([TypeVarParam(Name("T"))])
)

# (both) type CoolAlias[T: float] = int
print(make_type([Name("T", Name("float"))]))
print(
    make_type(
        [
            TypeVarParam(
                name=Name("T"),
                value=Name("float")
            )
        ]
    )
)

# type CoolAlias[*T] = int
print(
    make_type(
        [
            TypeVarTupleParam(
                Name("T"),
            )
        ]
    )
)

# type CoolAlias[**T] = int
print(
    make_type(
        [
            ParamSpecParam(
                Name("T"),
            )
        ]
    )
)

Generic Functions and Classes

FuncDef and ClassDef now accept an optional keyword argument type_params accepting Sequence[TypeParam]:

from gekkota import FuncDef, Name, PassStmt

typeparam = Name("T")

newfunc = FuncDef(
    name="newfunc",
    args=[Name("x", typeparam)],
    body=PassStmt(),
    rtype=typeparam,
    type_params=[typeparam],
)

print(newfunc)
"""
def newfunc[T](x: T) -> T: pass
"""

Assignment

For common assigment use Assignment:

from gekkota import Assignment, Name

a, b, c = map(Name, "abc")

# Assignment(targets: Sequence[AssignmentTarget] | AnnotatedTarget, value: Expression)

print(
    Assignment([a], b), # a = b
    Assignment([a.index(b)], c) # a[b] = c
    Assignment([a, b], c), # a = b = c
)

To annotate assignment (or just annotate a variable), use AnnotatedTarget:

from gekkota import Assignment, AnnotatedTarget, Name

a, b, c = map(Name, "abc")
D = Name("D")

# AnnotatedTarget(target: AssignmentTarget, annotation: Expression)
print(
    Assignment(AnnotatedTarget(a, D), b), # a: D = b
    Assignment(AnnotatedTarget(a.index(b), D), c) # a[b]: D = c
    Assignment([a, b], c), # a = b = c
)

For augmented assignment (e.g. +=) use AugmentedAssignment:

from gekkota import Assignment, Name

a, b, c = map(Name, "abc")

# AugmentedAssignment(target: AugAssignmentTarget, op: str, expression: Expression)

print(
    AugmentedAssignment(a, "+=", b), # a += b
    AugmentedAssignment(a.index(b), "*=", c) # a *= c
)

Control flow

For control flow you can use IfStmt, ElifStmt and ElseStmt:

from gekkota import Name, IfStmt, ElifStmt, ElseStmt, Code

a, b, c = map(Name, "abc")

# IfStmt(condition: Expression, body: Statement)
# ElifStmt(condition: Expression, body: Statement)
# ElseStmt(body: Statement)
code = Code([
    IfStmt(a, b),
    ElifStmt(b, a),
    ElseStmt(c)
])

print(code)
"""
if a: b
elif b: a
else: c
"""

Loops

Use ForStmt and WhileStmt for loops:

from gekkota import ForStmt, WhileStmt, Name

a, b, c = map(Name, "abc")

# ForStmt(target: Expression, iterator: Expression, body: Statement, *, is_async: bool = False)
print(
    ForStmt(a, b, c)
) # for a in b: c

# WhileStmt(condition: Expression, body: Statement)
print(
    WhileStmt(a, b)
) # while a: b

Functions

To render a function definition, you will need a FuncDef:

from gekkota import Name, FuncDef

a, b, c = map(Name, "abc")

# FuncDef(
#    name: str,
#    args: Sequence[FuncArg],
#    body: Statement,
#    *,
#    rtype: Optional[Expression] = None,
#    is_async: bool = False,
#    type_params: Sequence[TypeParam] = (),
# )

print(
    FuncDef(
        "cool_func",
        [a],
        b,
        rtype=c,
    )
) # def cool_func(a) -> c: b

New in 0.6: Now FuncDef accepts a type_params argument (check Generic Functions and Classes)

To provide a default value and/or annotations to arguments, use FuncArg:

from gekkota import Name, FuncDef, FuncArg, to_expression

a, b, c = map(Name, "abc")

# FuncDef(name: str, args: Sequence[FuncArg], body: Statement, *, rtype: Optional[Expression] = None, is_async: bool = False)
# FuncArg(
#     name: str,
#     annotation: Optional[Expression] = None,
#     default_value: Optional[Expression] = None,
#     *,
#     late_bound_default: bool = False,
# )
print(
    FuncDef(
        "cool_func",
        [
            FuncArg(
                "a",
                Name("int"),
                to_expression(0)
            )
        ],
        b,
        rtype=c,
    )
) # def cool_func(a: int = 0) -> c: b

New in version 0.6: support for PEP 671 (Draft) – Syntax for late-bound function argument defaults

Other argument types are:

  • StarArg(value: T = None): generates *value, * by default
  • DoubleStarArg(value): same as StarArg, but with **
  • Slash() is / (a mark of positional-only arguments in Python 3.8+)

Lambda functions are generated using LambDef:

from gekkota import Name, LambDef

a, b, c = map(Name, "abc")

# LambDef(args: Sequence[FuncArg], body: Expression)
print(
    LambDef(
        [a],
        b,
    )
) # lambda a: b

To decorate a function/class, use Decorated:

from gekkota import Name, FuncDef, Decorated

decorator = Name("decorator")
a, b, c = map(Name, "abc")

# Decorated(decorator: Expression, statement: ClassDef | FuncDef)
# FuncDef(name: str, args: Sequence[FuncArg], body: Statement, *, rtype: Optional[Expression] = None, is_async: bool = False)
print(
    Decorated(
        decorator,
        FuncDef(
            "cool_func",
            [a],
            b,
            rtype=c,
        )
    )
)
# @decorator
# def cool_func(a) -> c: b

Classes

To define a class, use ClassDef:

from gekkota import Name, ClassDef

a, b, c = map(Name, "abc")

# ClassDef(
#     name: str,
#     args: Sequence[CallArg | Expression],
#     body: Statement,
#     *,
#     type_params: Sequence[TypeParam] = (),
# )
print(
    ClassDef("MyClass1", [], a)
) # class MyClass1: a

print(
    ClassDef("MyClass2", [b], c)
) # class MyClass2(b): c

New in 0.6: Now ClassDef accepts a type_params argument (check Generic Functions and Classes)

Imports

To render imports, use ImportStmt and FromImportStmt:

from gekkota import Name, StarArg, ImportDots, ImportSource, ImportStmt, FromImportStmt, ImportAlias


# ImportStmt(names: Sequence[ImportAlias | Name | StarArg[None]])
print(
    ImportStmt([Name("a")])
) # import a

print(
    ImportStmt([Name("a"), Name("b")])
) # import a, b

# FromImportStmt(source: ImportSource | Name, names: Sequence[ImportAlias | Name | StarArg[None]])
# ImportAlias(name: Name, alias: Name | None = None)
print(
    FromImportStmt(
        Name("math"),
        [
            Name("cos"),
            ImportAlias(Name("sin"), Name("tan")) # we do a little trolling
        ]
    )
) # from math import cos, sin as tan

print(
    FromImportStmt(
        Name("gekkota"),
        [StarArg()]
    )
) # from gekkota import *

# ImportDots(length: int = 1)
print(
    FromImportStmt(
        ImportDots(),
        [StarArg()]
    )
) # from . import *

# ImportSource(parts: Sequence[str])
print(
    FromImportStmt(
        ImportSource(["", "values"]),
        [Name("Name")]
    )
) # from .values import Name

Exceptions

from gekkota import Name, TryStmt, ExceptStmt, FinallyStmt, Block

a, b, e = map(Name, "abe")

# TryStmt(body: Statement)
print(
    TryStmt(a)
) # try: a

# ExceptStmt(exceptions: Sequence[Expression] | None, alias: Name | None, body: Statement)
print(
    ExceptStmt(None, None, a)
) # except: a

print(
    ExceptStmt(None, None, Block([]))
)
# except:
#     pass

print(
    ExceptStmt([a], None, b)
) # except a: b

print(
    ExceptStmt([a], e, b)
) # except a as e: b

# FinallyStmt(body: Statement)
print(
    FinallyStmt(a)
) # finally: a

Context Managers

from gekkota import Name, WithStmt, WithTarget

a, b, e = map(Name, "abe")

# WithStmt(targets: Sequence[WithTarget | Expression], body: Statement, *, is_async: bool = False,)
print(
    WithStmt([a], b)
) # with a: b

# WithTarget(expression: Expression, alias: str | None = None)
print(
    WithStmt([WithTarget(a, "aaaa")], b)
) # with a as aaaa: b

print(
    WithStmt([a], b, is_async=True)
) # async with a: b

Pattern matching

This section is currently unfinished, check pattern_matching.py

Custom rendering

If your custom element can be meaningfully represented as a combination of existing elements, you can use a function instead of a class:

from gekkota import Expression

def Square(e: Expression) -> Expression:
    return e * e

This is a pretty obvious approach, but often it works best.


While being aimed at Python code generation, gekkota is pretty extensible, and can be used to render different things.
You can build custom renderables, statements, expressions, and so on.

The simplest example of a custom renderable would be:

from gekkota import Renderable, StrGen, Config


class RenderString(Renderable):
    """It renders whatever is passed to it"""

    def __init__(self, value: str):
        self.value = value

    def render(self, config: Config) -> StrGen:
        yield self.value

Let's suppose you want to render a custom expression: a custom sequence literal (obviously isn't valid in Python, but you need it for some reason).
Suppose your custom literal would be in form of <|value1, value2, ...|>.

You can extend SequenceExpr for that:

from gekkota import SequenceExpr, Name

class MyCoolSequence(SequenceExpr):
    parens = "<|", "|>"


seq = MyCoolSequence([Name("a"), Name("b")])

print(seq) # <|a,b|>

That's it, you're ready to render this literal (which, again, isn't valid in Python but anyway).

Or you could go further and write rendering by yourself (it's easier than it sounds):

from gekkota import Expression, Config


class MyCoolSequence(Expression):
    def __init__(self, values: Sequence[Expression]):
        self.values = values

    # could be rewritten to be simpler, check `Useful utils` section below
    def render(self, config: Config) -> StrGen:
        yield "<|"

        for i, item in enumerate(self.values):
            yield from item.render(config)

            if i + 1 < len(self.values): # no comma after last element
                yield ","
                yield " "

        yield "|>"

It's fairly easy, just render every part in the right order:

  • To render a string, use yield string
  • To render a Renderable, use yield from renderable.render(config)

Choosing a right base class

To choose a right base class, think in what context you want to use your renderable.
If there is a similar context in Python (e.g. your renderable is a block statement, like for or if), extend that class.

After choosing a right base class, check if it has a predefined render, maybe you won't need to write everything by yourself.
For example, with BlockStmt you need to provide render_head instead:

# that's the actual source from module, not an example

class BlockStmt(Statement):
    body: Statement

    def render_head(self, config: Config) -> StrGen:
        return NotImplemented

    def render(self, config: Config) -> StrGen:
        yield from self.render_head(config)
        yield ":"
        yield " "
        yield from self.body.render(config)

[...]

class ElseStmt(BlockStmt):
    def __init__(self, body: Statement):
        self.body = body

    def render_head(self, config: config) -> StrGen:
        yield "else"

Useful utils

gekkota.utils provides Utils class which is useful for custom renderables. For example, custom MyCoolSequence could be implemented as:

from gekkota import Expression, Utils


class MyCoolSequence(Expression):
    def __init__(self, values: Sequence[Expression]):
        self.values = values

    def render(self, config: config) -> StrGen:
        yield from Utils.wrap(
            ["<|", "|>"],
            Utils.comma_separated(self.values, config)
        )

Methods provided in Utils:

  • add_tab(generator: StrGen, config: Config) -> StrGen
    Adds indentation to a stream of tokens, using provided config
    For example, Utils.add_tab(Name("a").render(config), config) -> Iterable of [' ', 'a']

  • separated(separator: Sequence[str], renderables: Sequence[Renderable], config: Config) -> StrGen
    Inserts separator between renderables (and renders them in stream)
    For example: Utils.separated([",", " "], self.values, config) - inserts ", " between elements of self.values

  • separated_str(separator: Sequence[str], strings: Sequence[str], config: Config)
    Same as previous, but for str sequences

  • comma_separated(renderables: Sequence[Renderable], config: Config) -> StrGen
    Alias for Utils.separated([",", " "], renderables, config)

  • make_compact(generator: StrGen, config: Config) -> StrGen
    Filters all unneccessary whitespace from stream (doesn't respect config["compact"]). Config is unused at the moment, but provided for compatibility with future updates

  • wrap(parens: Sequence[str], generator: StrGen) -> StrGen
    Wraps a token stream with strings from parens array (should have 2 elements).
    In other words, inserts parens[0] at the start of the stream, and parens[1] at the end

wordstreamer compatibility

gekkota contains experimental wordstreamer compatibility layer to use gekkota objects in wordstreamer and vice versa.

Use gekkota.Renderable where wordstreamer.Renderable can be used:

from gekkota import WSRenderable, Literal

ws_compatible = WSRenderable(Literal(6)) # this is a wordstreamer.Renderable now

Use wordstreamer.Renderable where gekkota.Renderable can be used:

from gekkota import WStoG, Literal

from wordstreamer.startkit import Stringify

# create a class to define the type of renderable
class WSLiteral(WStoG, Literal):
    pass

some_number = Stringify(6)

print(WSLiteral(some_number) + Literal(8)) # "6 + 8"

Insert a wordstreamer.Renderable in a string (gekkota.Literal)

from gekkota import WSString, Literal

from wordstreamer.startkit import Stringify

some_number = Stringify(6)

print(WSString(some_number)) # '"""6"""'

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

gekkota-0.7.0.tar.gz (26.6 kB view details)

Uploaded Source

Built Distribution

gekkota-0.7.0-py3-none-any.whl (27.9 kB view details)

Uploaded Python 3

File details

Details for the file gekkota-0.7.0.tar.gz.

File metadata

  • Download URL: gekkota-0.7.0.tar.gz
  • Upload date:
  • Size: 26.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.2 CPython/3.9.19 Linux/6.5.0-1018-azure

File hashes

Hashes for gekkota-0.7.0.tar.gz
Algorithm Hash digest
SHA256 146ed86f7c593c56e1a92c28faf7dac04d1afaddcdd0a45f031f05f5e582eab7
MD5 a7b7e7b9912c1952f9f8219892fe95f2
BLAKE2b-256 3e43fd935738bccb33fb824ad9882267682530e3f135f1d2d5d871339315074e

See more details on using hashes here.

File details

Details for the file gekkota-0.7.0-py3-none-any.whl.

File metadata

  • Download URL: gekkota-0.7.0-py3-none-any.whl
  • Upload date:
  • Size: 27.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.8.2 CPython/3.9.19 Linux/6.5.0-1018-azure

File hashes

Hashes for gekkota-0.7.0-py3-none-any.whl
Algorithm Hash digest
SHA256 fef8b487e5f9411f4551726a076b8a2b438c66a4fc4d6905e49c144ff96dd72e
MD5 b479524d900b8b4b2c3a63493813e642
BLAKE2b-256 9e1b7a3d93df7d632f631013a7c9ad5faa86ddab320805d09a251a2554cdb9a7

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page