macro-polo 0.2.0

Rust-style macros for Python

macro-polo

Rust-style macros for Python

macro-polo brings Rust-inspired compile-time macros to Python. It's currently in very early alpha, but even if it ever gets a stable release, you probably shouldn't use it in any serious project. Even if you find a legitimate use case, the complete lack of tooling support almost definitely outweighs the benefits. That said, if you do decide to use it, I'd love to know why!

Usage

macro-polo is modular, and can be extended at multiple levels. See the API Documentation for more details.

The simplest way to use it is to add a coding: macro_polo comment to the top of your source file (in one of the first two lines). You can then declare and invoke macros using the macro_rules! syntax.

Example (bijection.py):

# coding: macro_polo
"""A basic demonstration of `macro_rules!`."""


macro_rules! bijection:
    [$($key:tt: $val:tt),* $(,)?]:
        (
            {$($key: $val),*},
            {$($val: $key),*}
        )


macro_rules! debug_print:
    [$($expr:tt)*]:
        print(
            stringify!($($expr)*), '=>', repr($($expr)*),
            file=__import__('sys').stderr,
        )


names_to_colors, colors_to_names = bijection! {
    'red': (1, 0, 0),
    'green': (0, 1, 0),
    'blue': (0, 0, 1),
}


debug_print!(names_to_colors)
debug_print!(colors_to_names)

debug_print!(names_to_colors['green'])
debug_print!(colors_to_names[(0, 0, 1)])

$ python3 examples/bijection.py
names_to_colors  => {'red': (1, 0, 0), 'green': (0, 1, 0), 'blue': (0, 0, 1)}
colors_to_names  => {(1, 0, 0): 'red', (0, 1, 0): 'green', (0, 0, 1): 'blue'}
names_to_colors ['green'] => (0, 1, 0)
colors_to_names [(0 ,0 ,1 )] => 'blue'

Viewing the generated code:

$ python3 -m macro_polo examples/bijection.py | ruff format -

names_to_colors, colors_to_names = (
    {'red': (1, 0, 0), 'green': (0, 1, 0), 'blue': (0, 0, 1)},
    {(1, 0, 0): 'red', (0, 1, 0): 'green', (0, 0, 1): 'blue'},
)
print(
    'names_to_colors',
    '=>',
    repr(names_to_colors),
    file=__import__('sys').stderr,
)
print(
    'colors_to_names',
    '=>',
    repr(colors_to_names),
    file=__import__('sys').stderr,
)
print(
    "names_to_colors ['green']",
    '=>',
    repr(names_to_colors['green']),
    file=__import__('sys').stderr,
)
print(
    'colors_to_names [(0 ,0 ,1 )]',
    '=>',
    repr(colors_to_names[(0, 0, 1)]),
    file=__import__('sys').stderr,
)

Other encodings

If you want to specify a text encoding, you can append it to macro_polo after a - or _, such as # coding: macro_polo-utf-16.

macro_rules!

macro_rules! declarations consist of one or more rules, where each rule consists of a matcher and a transcriber.

When the macro is invoked, it's input is compared to each matcher (in the order in which they were defined). If the input macthes, the capture variables are extracted and passed to the transcriber, which creates a new token sequence to replace the macro invocation.

This is the syntax for defining a macro_rules! macro:

macro_rules!:
    [matcher0]:
        transcriber0

    ...

    [matchern]:
        transcribern

macro_rules macros can be recursive by transcibing a new invocation to themselves. See braces_and_more.py for an example.

Matchers

The following constructs are supported in macro_rules! matchers:

Pattern	Description
$name:type	A capture variable.
$(pattern)sep?? $(pattern)sep?* $(pattern)sep?+	A pattern repeater. Matches pattern ≤1 (?), 0+ (*), or 1+ (+) times. If sep is present, it is a single-token separator that must match between each repitition. Capture variables inside repeaters become "repeating captures."
$[(pattern0)|...|(patternn)]	A union of patterns. Patterns are tried sequentially from left to right. All pattern variants must contain the same capture variable names at the same levels of repitition depth. The capture variable types, on the other hand, need not match.
$[!pattern]	A negative lookahead. Matches zero tokens if pattern fails to match. If pattern does match, the negative lookahead will fail.
$$	Matches a $ token.
$>	Matches an INDENT token.
$<	Matches a DEDENT token.
$^	Matches a NEWLINE token.

All other tokens are matched exactly (ex: 123 matches a NUMBER token with string '123')

Capture Variables

Capture variables are patterns that, when matched, bind the matching token(s) to a name (unless the name is _). They can then be used in a transcriber to insert the matched token(s) into the macro output.

Capture variables consist of a name and a type. The name can be any Python NAME token. The supported types are described in the table below:

type	Description
token	Matches any single token, except delimiters.
name	Matches a NAME token.
op	Matches an OP token, except delimiters.
number	Matches a NUMBER token.
string	Matches a STRING token.
tt	Matches a "token tree": either a single non-delimiter token, or a pair of (balanced) delimiters and all of the tokens between them.
null	Always matches zero tokens. Useful for counting repitions, or for filling in missing capture variables in union variants.

Transcribers

The following constructs are supported in macro_rules! transcribers:

Pattern	Description
$name	A capture variable substitution. Expands to the token(s) bound to name. If the corresponding capture variable appears within a repeater, the substitution must also be in a repeater at the same or greater nesting depth.
$(pattern)sep?*	A pattern repeater. There must be at least one repeating substitution in pattern, which determines how many times the pattern will be expanded. If pattern contains multiple repeating substitutions, they must repeat the same number of times (at the current nesting depth). If sep is present, it is a single-token separator that will be expanded before each repitition after the first.
$$	Expands to a $ token.
$>	Expands to an INDENT token.
$<	Expands to a DEDENT token.
$^	Expands to a NEWLINE token.

All other tokens are left unchanged.

Delimiters

Delimiters are pairs of tokens that enclose other tokens, and must always be balanced.

There are five types of delimiters:

• Parentheses ((, ))
• Brackets ([, ])
• Curly braces ({, })
• Indent/dedent
• f-strings

Note that f-strings come in many forms: f'...', rf"""...""", Fr'''...''', ....

Advanced Techniques

• Counting with null

  Let's write a macro that counts the number of token trees in its input. We'll do this by replacing each token tree with 1 + and then ending it of with a 0.

  We can write a recursive macro to recursively replace the first token tree, one-by-one:

  macro_rules! count_tts_recursive:
      [$t:tt $($rest:tt)*]:
          1 + count_tts_recursive!($($rest)*)
  
      []: 0
  

  Alternatively, we can use the null capture type to "count" the number of tts, and then emit the same number of 1 +s, all in one go:

  macro_rules! count_tts_with_null:
      [$($_:tt $counter:null)*]:
          $($counter 1 +)* 0
  

• Matching terminators with negative lookahead

  Let's write a macro that replaces ;s with newlines.

  # coding: macro_polo
  
  macro_rules! replace_semicolons_with_newlines_naive:
      [$($($line:tt)*);*]:
          $($($line)*)$^*
  
  replace_semicolons_with_newlines_naive! { if 1: print(1); if 2: print(2) }
  

  When we try to run this, however, we get a SyntaxError.

  If we use run macro_polo directly to check the code being emitted, we see something strange:

  $ python3 -m macro_polo negative_lookahead_naive.py
  if 1 :print (1 );if 2 :print (2 )
  

  The input is left completely unchanged!

  The reason for this is actually quite simple: the $line:tt capture variable matches the semicolon, so the the entire input is captured in a single repition (of the outer repeater). What we really want is for $line:tt to match anything except ;, which we can do with a negative lookahead:

  # coding: macro_polo
  
  macro_rules! replace_semicolons_with_newlines:
      [$($($[!;] $line:tt)*);*]:
          $($($line)*)$^*
  
  replace_semicolons_with_newlines! { if 1: print(1); if 2: print(2) }
  

  Notice the addition of $[!;] before $line:tt. Now when we run this code, we get the output we expected:

  $ python3 examples/negative_lookahead.py
  1
  2
  

Procedural Macros

For more complex macros, you can define a macro as a Python function that takes a sequence of tokens as input and returns a new sequence of tokens as output. These are referred to as "procedural macros" or "proc macros".

There are three types of procedural macros:

1. function-style:

   signature: (tokens: Sequence[Token]) -> Sequence[Token]

   Invoked the same way as macro_rules macros:

   name!(input), name![input], name!{input}, or

   name!:
       input
   

   When invoked, the function is called with the token sequence passed as input.

2. module-level:

   signature: (parameters: Sequence[Token], tokens: Sequence[Token]) -> Sequence[Token]

   Invoked with ![name(parameters)] or ![name] (equivalent to ![name()]). Module-level macro invocations must come before all other code (with the exception of a docstring), and must each appear on their own line.

   When invoked, parameters is the token sequence following name, excluding the outer parentheses. tokens is the tokenized module starting from the line immediately following the invocation.

3. decorator-style:

   signature: (parameters: Sequence[Token], tokens: Sequence[Token]) -> Sequence[Token]

   Invoked with @![name(parameters)] or @![name] (equivalent to @![name()]). Decorator-style macro invocations must immediately precede a "block", defined as either a single newline-terminated line, or a line followed by an indented block.

   When invoked, parameters is the token sequence following name, excluding the outer parentheses. tokens is the tokenized block immediately following the invocation.

Exporting and Importing Proc Macros

An important thing to know about proc macros is that they cannot be invoked in the same module in which they are defined.

Instead, you use one of the three predefined decorator macros function_macro, module_macro, and decorator_macro to mark a macro for export. You can then import it using the predefined import module macro.

All three export macros take an optional name parameter as an alternative name to use when exporting the macro. By default the name of the function is used.

Example braces.py:

# coding: macro_polo
"""An example of a module proc macro that adds braces-support to Python."""

import token

from macro_polo import Token


@![module_macro]
def braces(parameters, tokens):
    """Add braces support to a Python module.

    The following sequences are replaced:
    - `{:` becomes `:` followed by INDENT
    - `:}` becomes DEDENT
    - `;` becomes NEWLINE
    """
    output = []
    i = 0
    while i < len(tokens):
        match tokens[i:i+2]:
            case Token(token.OP, '{'), Token(token.OP, ':'):
                output.append(Token(token.OP, ':'))
                output.append(Token(token.INDENT, ''))
                i += 2
            case Token(token.OP, ':'), Token(token.OP, '}'):
                output.append(Token(token.DEDENT, ''))
                i += 2
            case Token(token.OP, ';'), _:
                output.append(Token(token.NEWLINE, '\n'))
                i += 1
            case _:
                output.append(tokens[i])
                i += 1

    return output

We can then import and invoke our braces macro:

# coding: macro_polo
"""An example of using the `import` macro and invoking a module macro."""
![import(braces)]
![braces]


for i in range(5) {:
    print('i =', i);
    if i % 2 == 0 {:
        print(i, 'is divisible by 2')
    :}
:}

$ python3 examples/proc_macros.py/uses_braces.py
i = 0
0 is divisible by 2
i = 1
i = 2
2 is divisible by 2
i = 3
i = 4
4 is divisible by 2

Practically, you'll probably want to use macro_polo's lower-level machinary, instead of re-implementing matching and transcribing.

The import macro

We saw an example of importing a macro from another module. By default the import macro will import all macros (including macro_rules macros) from the target module. If you want to import specific macros, you can use the alternative ![import(x, y from z)] syntax.

One quirk of the import macro is that macro_rules imports are transitive (if module b import a macro_rules macro from module a, and then module c imports b, the macro_rules macro from a will be imported into c.) Proc macros, however, are not transitive.

API Documentation

WIP