Parsing

Parsing (better: lexing) text always starts in a lexicon, which is called the root lexicon. The rules in that lexicon are tried one by one. As soon as there is a match, a Token is generated with the matching text, the position of the text and the action that was specified in the rule. And if a target was specified, parsing continues in a different lexicon.

The tokens are put in a tree structure. Every active lexicon creates a Context list that holds the tokens and child contexts. If a target pops back to a previous lexicon, the previous context becomes the current one again.

All tokens and contexts point to their parents, so it is possible to manipulate and query the tree structure in various ways.

The structure of the tree is built by the TreeBuilder, see the tree and the treebuilder module. At the root is the Context carrying the root lexicon. The root context contains Tokens and/or other Contexts.

The TreeBuilder is capable of tokenizing the text in a background thread and also to rebuild just a changed part of the text, smartly reusing earlier generated tokens if possible.

Iterating and Querying

Both Token and Context have many methods for iterating over the tree, for getting at the parent, child or sibling nodes. Context has various find() methods to quickly find a token or context at a certain position in the text.

Using the Context.query property you can build XPath-like chains of filtering queries to quickly find tokens or contexts based on text, action or lexicon. This is described in the query module.

Example

Here are some examples of how to create a Language class and then use it:

import livelex

from livelex import (
    Language, lexicon,
    words, bygroup, bymatch, bytext,
    default_action,
    default_target,
    skip,
    tomatch,
)

class MyLang(Language):
    """A Language represents a set of Lexicons comprising a specific language.

    A Language is never instantiated. The class itself serves as a namespace
    and can be inherited from.



    """

    @lexicon(re_flags=0)
    def root(cls):
        yield r'"', "string", cls.string
        yield r'\(', "paren", cls.parenthesized
        yield r'\d+', "number"
        yield r'%', "comment", cls.comment
        yield r'[,.!?]', "punctuation"
        yield r'\w+', "word"

    @lexicon
    def string(cls):
        yield r'\\[\\"]', 'string escape'
        yield r'"', "string", -1
        yield default_action, "string"

    @lexicon(re_flags=re.MULTILINE)
    def comment(cls):
        yield r'$', "comment", -1
        yield r'XXX|TODO', "todo"
        yield default_action, "comment"

    @lexicon
    def parenthesized(cls):
        yield r'\)', "paren", -1
        yield from cls.root()


s = r"""
This is (an example) text with 12 numbers
and "a string with \" escaped characters",
and a % comment that TODO lasts until the end
of the line.
"""


>>> import livelex
>>> tree = livelex.root(MyLang.root, s)
>>> tree.dump()
<Context MyLang.root at 1-144 (20 children)>
 ├╴<Token 'This' at 1 (word)>
 ├╴<Token 'is' at 6 (word)>
 ├╴<Token '(' at 9 (paren)>
 ├╴<Context MyLang.parenthesized at 10-21 (3 children)>
 │  ├╴<Token 'an' at 10 (word)>
 │  ├╴<Token 'example' at 13 (word)>
 │  ╰╴<Token ')' at 20 (paren)>
 ├╴<Token 'text' at 22 (word)>
 ├╴<Token 'with' at 27 (word)>
 ├╴<Token '12' at 32 (number)>
 ├╴<Token 'numbers' at 35 (word)>
 ├╴<Token 'and' at 43 (word)>
 ├╴<Token '"' at 47 (string)>
 ├╴<Context MyLang.string at 48-84 (4 children)>
 │  ├╴<Token 'a string with ' at 48 (string)>
 │  ├╴<Token '\\"' at 62 (string escape)>
 │  ├╴<Token ' escaped characters' at 64 (string)>
 │  ╰╴<Token '"' at 83 (string)>
 ├╴<Token ',' at 84 (punctuation)>
 ├╴<Token 'and' at 86 (word)>
 ├╴<Token 'a' at 90 (word)>
 ├╴<Token '%' at 92 (comment)>
 ├╴<Context MyLang.comment at 93-131 (3 children)>
 │  ├╴<Token ' comment that ' at 93 (comment)>
 │  ├╴<Token 'TODO' at 107 (todo)>
 │  ╰╴<Token ' lasts until the end' at 111 (comment)>
 ├╴<Token 'of' at 132 (word)>
 ├╴<Token 'the' at 135 (word)>
 ├╴<Token 'line' at 139 (word)>
 ╰╴<Token '.' at 143 (punctuation)>
>>> tree.find_token(50)
<Token 'a string with ' at 48 (string)>
>>> tree.find_token(50).parent
<Context MyLang.string at 48-84 (4 children)>

>>> d = livelex.Document(MyLang.root, s)
>>> d
<Document '\nThis is (an example) text w...'>

>>> print(d.text())

This is (an example) text with 12 numbers
and "a string with \" escaped characters",
and a % comment that TODO lasts until the end
of the line.

>>> d[50:56]
'string'
>>> with d:
...  d[9:12] = '(a "much longer'
...  d[20:20] = '"'
...
>>> print(d.text())

This is (a "much longer example") text with 12 numbers
and "a string with \" escaped characters",
and a % comment that TODO lasts until the end
of the line.

>>> d.get_root()[3].dump()
<Context MyLang.parenthesized at 10-34 (4 children)>
 ├╴<Token 'a' at 10 (word)>
 ├╴<Token '"' at 12 (string)>
 ├╴<Context MyLang.string at 13-33 (2 children)>
 │  ├╴<Token 'much longer example' at 13 (string)>
 │  ╰╴<Token '"' at 32 (string)>
 ╰╴<Token ')' at 33 (paren)>