cs.lex 20190220

Lexical analysis functions, tokenisers.

Lexical analysis functions, tokenisers.

An arbitrary assortment of lexical and tokenisation functions useful for writing recursive descent parsers, of which I have several.

Generally the get_* functions accept a source string and an offset (usually optional, default 0) and return a token and the new offset, raising ValueError on failed tokenisation.

Function as_lines(chunks, partials=None)

Generator yielding complete lines from arbitrary pieces of text from the iterable chunks.

After completion, any remaining newline-free chunks remain in the partials list; this will be unavailable to the caller unless the list is presupplied.

Function get_chars(s, offset, gochars)

Scan the string s for characters in gochars starting at offset. Return (match, new_offset).

Function get_decimal(s, offset=0)

Scan the string s for decimal characters starting at offset. Return (dec_string, new_offset).

Function get_decimal_or_float_value(s, offset=0)

Fetch a decimal or basic float (nnn.nnn) value from the str s at offset. Return (value, new_offset).

Function get_decimal_value(s, offset=0)

Scan the string s for a decimal value starting at offset. Return (value, new_offset).

Function get_delimited(s, offset, delim)

Collect text from the string s from position offset up to the first occurence of delimiter delim; return the text excluding the delimiter and the offset after the delimiter.

Function get_dotted_identifier(s, offset=0, **kw)

Scan the string s for a dotted identifier (by default an ASCII letter or underscore followed by letters, digits or underscores) with optional trailing dot and another dotted identifier, starting at offset (default 0). Return (match, new_offset).

Note: the empty string and an unchanged offset will be returned if there is no leading letter/underscore.

Function get_envvar(s, offset=0, environ=None, default=None, specials=None)

Parse a simple environment variable reference to $varname or $x where "x" is a special character.

Parameters:

• s: the string with the variable reference
• offset: the starting point for the reference
• default: default value for missing environment variables; if None (the default) a ValueError is raised
• environ: the environment mapping, default os.environ
• specials: the mapping of special single character variables

Function get_hexadecimal(s, offset=0)

Scan the string s for hexadecimal characters starting at offset. Return hex_string, new_offset.

Function get_hexadecimal_value(s, offset=0)

Scan the string s for a hexadecimal value starting at offset. Return (value, new_offset).

Function get_identifier(s, offset=0, alpha='abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ', number='0123456789', extras='_')

Scan the string s for an identifier (by default an ASCII letter or underscore followed by letters, digits or underscores) starting at offset (default 0). Return (match, new_offset).

Note: the empty string and an unchanged offset will be returned if there is no leading letter/underscore.

Parameters:

• s: the string to scan
• offset: the starting offset, default 0.
• alpha: the characters considered alphabetic, default string.ascii_letters.
• number: the characters considered numeric, default string.digits.
• extras: extra characters considered part of an identifier, default '_'.

Function get_nonwhite(s, offset=0)

Scan the string s for characters not in string.whitespace starting at offset (default 0). Return (match, new_offset).

Function get_other_chars(s, offset=0, stopchars=None)

Scan the string s for characters not in stopchars starting at offset (default 0). Return (match, new_offset).

Function get_qstr(s, offset=0, q='"', environ=None, default=None, env_specials=None)

Get quoted text with slosh escapes and optional environment substitution.

Parameters:

• s: the string containg the quoted text.
• offset: the starting point, default 0.
• q: the quote character, default '"'. If q is set to None, do not expect the string to be delimited by quote marks.
• environ: if not None, also parse and expand $envvar references.
• default: passed to get_envvar

Function get_qstr_or_identifier(s, offset)

Parse a double quoted string or an identifier.

Function get_sloshed_text(s, delim, offset=0, slosh='\\', mapper=<function slosh_mapper at 0x10c07b598>, specials=None)

Collect slosh escaped text from the string s from position offset (default 0) and return the decoded unicode string and the offset of the completed parse.

Parameters:

• delim: end of string delimiter, such as a single or double quote.
• offset: starting offset within s, default 0.
• slosh: escape character, default a slosh ('').
• mapper: a mapping function which accepts a single character and returns a replacement string or None; this is used the replace things such as '\t' or '\n'. The default is the slosh_mapper function, whose default mapping is SLOSH_CHARMAP.
• specials: a mapping of other special character sequences and parse functions for gathering them up. When one of the special character sequences is found in the string, the parse function is called to parse at that point. The parse functions accept s and the offset of the special character. They return the decoded string and the offset past the parse.

The escape character slosh introduces an encoding of some replacement text whose value depends on the following character. If the following character is:

• the escape character slosh, insert the escape character.
• the string delimiter delim, insert the delimiter.
• the character 'x', insert the character with code from the following 2 hexadecimal digits.
• the character 'u', insert the character with code from the following 4 hexadecimal digits.
• the character 'U', insert the character with code from the following 8 hexadecimal digits.
• a character from the keys of mapper

Function get_tokens(s, offset, getters)

Parse the string s from position offset using the supplied tokenise functions getters; return the list of tokens matched and the final offset.

Parameters:

• s: the string to parse.
• offset: the starting position for the parse.
• getters: an iterable of tokeniser specifications.

Each tokeniser specification is either:

• a callable expecting (s, offset) and returning (token, new_offset)
• a literal string, to be matched exactly
• a tuple or list with values (func, args, kwargs); call func(s, offset, *args, **kwargs)
• an object with a .match method such as a regex; call getter.match(s, offset) and return a match object with a .end() method returning the offset of the end of the match

Function get_uc_identifier(s, offset=0, number='0123456789', extras='_')

Scan the string s for an identifier as for get_identifier(), but require the letters to be uppercase.

Function get_white(s, offset=0)

Scan the string s for characters in string.whitespace starting at offset (default 0). Return (match, new_offset).

Function hexify(bs)

A Python 2 flavour of binascii.hexlify.

Function htmlify(s, nbsp=False)

Convert a string for safe transcription in HTML.

Parameters:

• s: the string
• nbsp: replaces spaces with " " to prevent word folding, default False.

Function htmlquote(s)

Quote a string for use in HTML.

Function is_dotted_identifier(s, offset=0, **kw)

Test if the string s is an identifier from position offset onward.

Function is_identifier(s, offset=0, **kw)

Test if the string s is an identifier from position offset onward.

Function isUC_(s)

Check that a string matches ^[A-Z][A-Z_0-9]*$.

Function jsquote(s)

Quote a string for use in JavaScript.

Function lastlinelen(s)

The length of text after the last newline in a string.

(Initially used by cs.hier to compute effective text width.)

Function match_tokens(s, offset, getters)

Wrapper for get_tokens which catches ValueError exceptions and returns (None, offset).

Function parseUC_sAttr(attr)

Take an attribute name and return (key, is_plural).

'FOO' returns (FOO, False). 'FOOs' or 'FOOes' returns ('FOO', True). Otherwise return (None, False).

Function phpquote(s)

Quote a string for use in PHP code.

Function skipwhite(s, offset=0)

Convenience routine for skipping past whitespace; returns the offset of the next nonwhitespace character.

Function slosh_mapper(c, charmap=None)

Return a string to replace backslash-c, or None.

Function stripped_dedent(s)

Slightly smarter dedent which ignores a string's opening indent.

Strip the supplied string s. Pull off the leading line. Dedent the rest. Put back the leading line.

Example:

>>> def func(s):
...   """ Slightly smarter dedent which ignores a string's opening indent.
...       Strip the supplied string `s`. Pull off the leading line.
...       Dedent the rest. Put back the leading line.
...   """
...   pass
...
>>> from cs.lex import stripped_dedent
>>> print(stripped_dedent(func.__doc__))
Slightly smarter dedent which ignores a string's opening indent.
Strip the supplied string `s`. Pull off the leading line.
Dedent the rest. Put back the leading line.

Function strlist(ary, sep=', ')

Convert an iterable to strings and join with ", ".

Function tabpadding(padlen, tabsize=8, offset=0)

Compute some spaces to use a tab padding at an offfset.

Function texthexify(bs, shiftin='[', shiftout=']', whitelist=None)

Transcribe the bytes bs to text using compact text runs for some common text values.

This can be reversed with the untexthexify function.

This is an ad doc format devised to be compact but also to expose "text" embedded within to the eye. The original use case was transcribing a binary directory entry format, where the filename parts would be somewhat visible in the transcription.

The output is a string of hexadecimal digits for the encoded bytes except for runs of values from the whitelist, which are enclosed in the shiftin and shiftout markers and transcribed as is. The default whitelist is values of the ASCII letters, the decimal digits and the punctuation characters '_-+.,'. The default shiftin and shiftout markers are '[' and ']'.

Example:

>>> texthexify(b'&^%&^%abcdefghi)(*)(*')
'265e25265e25[abcdefghi]29282a29282a'

Parameters:

• bs: the bytes to transcribe
• shiftin: Optional. The marker string used to indicate a shift to direct textual transcription of the bytes, default: '['.
• shiftout: Optional. The marker string used to indicate a shift from text mode back into hexadecimal transcription, default ']'.
• whitelist: an optional bytes or string object indicating byte values which may be represented directly in text; string objects are converted to hexify() and texthexify() output strings may be freely concatenated and decoded with untexthexify(). The default value is the ASCII letters, the decimal digits and the punctuation characters '_-+.,'.

Function unctrl(s, tabsize=8)

Return the string s with TABs expanded and control characters replaced with printable representations.

Function untexthexify(s, shiftin='[', shiftout=']')

Decode a textual representation of binary data into binary data.

This is the reverse of the texthexify function.

Outside of the shiftin/shiftout markers the binary data are represented as hexadecimal. Within the markers the bytes have the values of the ordinals of the characters.

Example:

>>> untexthexify('265e25265e25[abcdefghi]29282a29282a')
b'&^%&^%abcdefghi)(*)(*'

Parameters:

• s: the string containing the text representation.
• shiftin: Optional. The marker string commencing a sequence of direct text transcription, default '['.
• shiftout: Optional. The marker string ending a sequence of direct text transcription, default ']'.