dbrownell-parserlib 0.8.4

Functionality useful when creating parsers.

Project:

Package:

Development:

Contents

• Overview
• Installation
• Development
• Additional Information
• License

Overview

dbrownell_ParserLib is a Python library that provides utility functionality for creating parsers, with a primary focus on ANTLR (Another Tool for Language Recognition) integration. The library simplifies the process of building parsers that produce strongly-typed abstract syntax trees (ASTs) with comprehensive source location tracking.

Key Features

• ANTLR Integration: Streamlined wrapper around ANTLR for Python, handling grammar compilation and parser generation
• AST Representation: Expression-based AST nodes with automatic parent-child relationship management
• Source Location Tracking: Every AST node tracks its source file location (filename, line, column ranges) for detailed error reporting
• Visitor Pattern: Flexible visitor pattern implementation with fine-grained traversal control
• Error Handling: Rich error objects that associate messages with source locations and extract Python exception tracebacks
• Multi-threaded Parsing: Built-in support for parsing multiple files concurrently
• Workspace Support: Parse entire directory trees with customizable file filtering
• Whitespace Handling: Mixins for both insignificant and significant whitespace grammars (e.g., Python-like indentation)

How to use dbrownell_ParserLib

The typical workflow for creating a parser with this library involves the following steps. For detailed examples, see the test files in the repository, particularly the end-to-end tests that demonstrate complete parsing workflows.

1. Define an ANTLR Grammar

Create a .g4 grammar file defining your language syntax:

grammar Calculator;

expr : expr ('*'|'/') expr   # BinaryOp
     | expr ('+'|'-') expr   # BinaryOp
     | INT                   # Number
     ;

INT : [0-9]+ ;
WS : [ \t\n\r]+ -> skip ;

2. Build the Grammar

Use BuildAntlrGrammar to generate the lexer and parser:

from pathlib import Path

from dbrownell_ParserLib import BuildAntlrGrammar


BuildAntlrGrammar(
    dm,  # DoneManager instance
    Path("Calculator.g4"),
    Path("output_dir"),
)

3. Create Custom Visitor

Implement a visitor that converts ANTLR parse trees to Expression objects:

from dataclasses import dataclass

from dbrownell_ParserLib import (
    Expression,
    InsignificantWhitespaceAntlrVisitorMixin,
    TerminalExpression,
)

from CalculatorVisitor import CalculatorVisitor as GeneratedVisitor


@dataclass(eq=False)
class BinaryExpression(Expression):
    left: Expression
    operator: TerminalExpression[str]
    right: Expression

    def _GenerateAcceptDetails(self):
        yield "left", self.left
        yield "operator", self.operator
        yield "right", self.right


class CalculatorVisitor(InsignificantWhitespaceAntlrVisitorMixin, GeneratedVisitor):
    def visitExpr(self, ctx):
        children = self.GetChildren(ctx)
        assert len(children) == 2, children

        self._stack.append(
            BinaryExpression(
                self.CreateRegion(ctx),
                children[0],
                TerminalExpression[str](self.CreateRegion(ctx.children[1]), ctx.children[1].getText()),

                children[1],
            )
        )

    def visitNumber(self, ctx):
        self._stack.append(
            TerminalExpression(self.CreateRegion(ctx), int(ctx.getText()))
        )

4. Create Parser Instance

Use CreateAntlrParser to build a callable parser:

from dbrownell_ParserLib import CreateAntlrParser


parser = CreateAntlrParser(
    CalculatorLexer,
    CalculatorParser,
    CalculatorVisitor,
    lambda p: p.expr(),  # Entry point rule
)

5. Parse Files

Parse single files, multiple files, or entire workspaces:

from dbrownell_ParserLib import Error


# Parse a single file
results = parser(dm, Path("input.txt"), None)

# Parse multiple files
results = parser(dm, [Path("file1.txt"), Path("file2.txt")], None)

# Check results
for filepath, result in results.items():
    if isinstance(result, Error):
        print(f"Parse error in {filepath}: {result}")
    else:
        # result is the visitor containing the parsed AST
        ast = result._stack[0]
        # Process the AST

6. Traverse the AST

Use the visitor pattern to process your AST:

from contextlib import contextmanager

from dbrownell_ParserLib import ExpressionVisitorHelper, VisitResult


class EvaluatorVisitor(ExpressionVisitorHelper):
    @contextmanager
    def OnBinaryExpression(self, expression):
        # Process binary expressions
        yield VisitResult.Continue


ast.Accept(EvaluatorVisitor())

Installation

Installation Method	Command
Via uv	uv add dbrownell_ParserLib
Via pip	pip install dbrownell_ParserLib

Verifying Signed Artifacts

Artifacts are signed and verified using py-minisign and the public key in the file ./minisign_key.pub.

To verify that an artifact is valid, visit the latest release and download the .minisign signature file that corresponds to the artifact, then run the following command, replacing <filename> with the name of the artifact to be verified:

uv run --with py-minisign python -c "import minisign; minisign.PublicKey.from_file('minisign_key.pub').verify_file('<filename>'); print('The file has been verified.')"

Development

Please visit Contributing and Development for information on contributing to this project.

Additional Information

Additional information can be found at these locations.

Title	Document	Description
Code of Conduct	CODE_OF_CONDUCT.md	Information about the norms, rules, and responsibilities we adhere to when participating in this open source community.
Contributing	CONTRIBUTING.md	Information about contributing to this project.
Development	DEVELOPMENT.md	Information about development activities involved in making changes to this project.
Governance	GOVERNANCE.md	Information about how this project is governed.
Maintainers	MAINTAINERS.md	Information about individuals who maintain this project.
Security	SECURITY.md	Information about how to privately report security issues associated with this project.

License

dbrownell_ParserLib is licensed under the MIT license.