Skip to main content

Library for automatic construction of lexers

Project description

About LexBuilder:

LexBuilder is a library for automatically building a lexer in Python and C++. In the future, the library will be able to build lexers in major programming languages such as Golang, Java/Kotlin, etc.

About Syntax:

In order for the library to generate a Lexer.py or Lexer.h file, you need to pass a list of tokens to the Builder class or use its .add_token() method. To declare a token, you need to import the Token() class from the Builder() class. You need to pass the token name and its value to the Token() class. After that, add all the tokens we created to a list and pass it as an argument to the PyBuilder() or CppBuilder class. By default, the lexer contains the tokens:

INT_NUMBER = "INT_NUMBER"
FLOAT_NUMBER = "FLOAT_NUMBER"
PLUS = "PLUS"
MINUS = "MINUS"
STRING = "STRING"
MULTIPLY = "MULTIPLY"
DIVIDE = "DIVIDE"
VAR = "VAR"
ASSIGN = "ASSIGN"
EOF = "EOF"

Example of creating a list of tokens:

from LexBuilder.Builder import Token

PRINT = Token("PRINT", "print")
INPUT = Token("INPUT", "input")

tokens = [PRINT, INPUT]

Python Example:

Generate Lexer:

from LexBuilder.Builder import PyBuilder, Token

PRINT = Token("PRINT", "print")
tokens = [PRINT]

lexer = PyBuilder(tokens)

lexer.add_token(Token("INPUT", "input"))
lexer.add_token(Token("LPAREN", "("))
lexer.add_token(Token("RPAREN", ")"))

lexer.build()

Use Lexer:

from Lexer import *


code = 'print("Hello, world!")'
lexer = Lexer(code)

token = lexer.get_next_token()
print(token)

while token.type != EOF:
    token = lexer.get_next_token()
    print(token)
Token(PRINT, "print")
Token(LPAREN, "(")
Token(STRING, "Hello, world!")
Token(RPAREN, ")")
Token(EOF, None)

C++ Example:

Generate Lexer:

from LexBuilder.Builder import CppBuilder, Token

PRINT = Token("PRINT", "print")
tokens = [PRINT]

lexer = CppBuilder(tokens)

lexer.add_token(Token("INPUT", "input"))
lexer.add_token(Token("LPAREN", "("))
lexer.add_token(Token("RPAREN", ")"))

lexer.build()

Use Lexer:

#include "Lexer.h"

using namespace std;


int main() {
    string code = "5 + 5";
    Lexer lexer(code);
    Token current_token = lexer.get_next_token();
    cout << current_token << endl;
    while (current_token.type != EOF_TOKEN) {
        current_token = lexer.get_next_token();
        cout << current_token << endl;
    }
}
Token(INT_NUMBER, 5)
Token(PLUS, +)
Token(INT_NUMBER, 5)
Token(EOF, )

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

lexbuilder-1.3.0.tar.gz (5.1 kB view details)

Uploaded Source

Built Distribution

LexBuilder-1.3.0-py3-none-any.whl (6.2 kB view details)

Uploaded Python 3

File details

Details for the file lexbuilder-1.3.0.tar.gz.

File metadata

  • Download URL: lexbuilder-1.3.0.tar.gz
  • Upload date:
  • Size: 5.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.3

File hashes

Hashes for lexbuilder-1.3.0.tar.gz
Algorithm Hash digest
SHA256 5ef2e3c9ef12058df2e90f001d90d46443e590caa30fdefe12355d7d6ebb1c67
MD5 ebcede1d00a1d5f67caa7753c35e49e2
BLAKE2b-256 5fc6751a88f22aef6de464f7d6115e89ebdfe02e832ddd85c829b7b6a4d2670e

See more details on using hashes here.

File details

Details for the file LexBuilder-1.3.0-py3-none-any.whl.

File metadata

  • Download URL: LexBuilder-1.3.0-py3-none-any.whl
  • Upload date:
  • Size: 6.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.12.3

File hashes

Hashes for LexBuilder-1.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 3a77574d8d8c982f495e3451d06786a746adfd7df4e5254ad685e18e00ce7ace
MD5 9c11a25fd245c473cb63e48a88638b99
BLAKE2b-256 42149664d8869b6d44b504dfe64b2dcba11fb72f9178abea4499b3bbe23c0686

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page