Skip to main content

A python implementation of a lexical analyzer which supports full scan, state based lexing and lookahead

Project description


A python implementation of a lexical analyzer which supports full scan, state based lexing and lookahead

Build Status PyPI - Python Version PyPI version PyPI license

Warning This is not a GENERATOR like classical lexer is. It does not produce any python code. It's a simple plain scanner of the given input string and tokenizer into given set of tokens by matching regular expressions. Thus, at runtime you can change the token definition and use one same code for any token set


Install in your project with pip:

pip install pylexer


An example use:

from pylexer import PyLexer
config = {
	'\\s' :'',
    '\\d+' :'number',
    '\\+' : 'plus',
    '-': 'minus',
    '\\*' : 'mul',
    '/' : 'div',
#Static Scan method that returns list of tokens
tokens = PyLexer.scan(config, '2 + 3')
map(lambda x:x.get_name(), tokens)

#PyLexer Config is a dict, so you can also use it like
lexer = PyLexer()
lexer.set_input('2 + 3')
while lexer.get_look_ahead():

Token Definition

Tokens are defined with TokenDefinition class that holds token name and regular expression. Token name can be empty, and in that case lexer will ignore/skip such tokens

Lexer Configuration

The lexer configuration holds a list of all token definitions. With LexerDictConfig it can be easily created from an array where keys are regular expressions and values are names of tokens

Full scan

Pylexer's static scan method can be used to scan given input string and returns a list of tokens, Pylexer can also be used to walk through scanned tokens with single look ahead


MIT license. See for more information.


Pylexer is inspired from PHP's Lexer( and takes code heavily from doctrine API, all credits due with Milos Tomic

Project details

Release history Release notifications

This version


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pylexer, version 1.0.0
Filename, size File type Python version Upload date Hashes
Filename, size pylexer-1.0.0-py3-none-any.whl (7.8 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size pylexer-1.0.0.tar.gz (5.9 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page