Skip to main content

Lexer generation

Project description

Lexit

Lexit is an open source lexer generator written in Python3.6 using new features like NamedTuple, type hinting and __init_subclass__ hook.

JSON lexer example

from lexit import Lexer


class JsonLexer(Lexer):
    NUMBER = r'-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][+-]?\d+)?'
    STRING = r'"(\\\"|\\\\|[^"\n])*?"i?'
    L_BRACE = r'{'
    R_BRACE = r'}'
    L_BRACKET = r'\['
    R_BRACKET = r'\]'
    TRUE = r'true'
    FALSE = r'false'
    NULL = r'null'
    COMMA = r','
    COLON = r':'

    ignore = r'\s+'


tokens = list(JsonLexer.lex('{"hello": "world"}'))

Requirements

  • The only requirement is Python3.6+
  • For testing the pytest library is used

Installation

pip install lexit

Error handling

try:
    tokens = list(JsonLexer.lex('${"hello": "world"}'))
except LexerError as e:
    print(e.pretty())
    exit(1)

# Will produce the following output:

No match for character '$' in line 1 column 1
${"hello": "world"}
^

Design decisions

  • Should be easy to use
  • Longest match priority (++ always wins over + despite of the order in which the tokens are defined in the lexer class)
  • Self-describing errors for humans (it's should be obvious what happened and when)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lexit-0.0.1.tar.gz (3.3 kB view details)

Uploaded Source

Built Distribution

lexit-0.0.1-py3-none-any.whl (3.4 kB view details)

Uploaded Python 3

File details

Details for the file lexit-0.0.1.tar.gz.

File metadata

  • Download URL: lexit-0.0.1.tar.gz
  • Upload date:
  • Size: 3.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for lexit-0.0.1.tar.gz
Algorithm Hash digest
SHA256 400119d224e272fda8af809100de074f7d81bd292d5a2af3212d8f64b85ee4e7
MD5 2d57120df102a8d106e8dcc21d7e47cf
BLAKE2b-256 772deeed45dd47b608d0c1f33b3a2b6f48376df96d100e7836d51b08884bf60f

See more details on using hashes here.

File details

Details for the file lexit-0.0.1-py3-none-any.whl.

File metadata

File hashes

Hashes for lexit-0.0.1-py3-none-any.whl
Algorithm Hash digest
SHA256 abf3eff2630d7a440cb222b4db4fd07e04aefb9ef3d7313cd44d7a1e912124df
MD5 12fada084394153c16fece2f5d4f32ee
BLAKE2b-256 eaf047d7c907918bac5e2882a2815130a62a447814476e093df412390c487ca0

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page