Skip to main content

No project description provided

Project description

PyPascalTokenizer
=================

Module for Python 3. It does tokenizing of Pascal code (it's planned to support full Delphi and FreePascal syntax).

API
===

Token struct
------------

Token is saved as 4-tuple (text, begin, end, final):

* text: string with token text
* begin: position of token start, tuple (y, x), where y - 0-based line index and x - 0-based character index in line
* end: position after token end, tuple (y, x)
* final: bool, True if it was last token

Class PasTokenizer
------------------

* __init__(lines): params: list of source code strings
* get_next(): get next token and change end pos
* read_next(): get next token, but don't change end pos
* is_ended(): check if text ended

Class PasTokenizerStack
-----------------------

* __init__(lines, comments=True): params: list of source code strings; "comments" allows to get also comment-tokens (otherwise tokenizer skips them)
* push(s): push token to stack
* pop(): pop (get and delete) token from stack
* read_last(): read (get but don't delete) top token from stack
* is_ended(): check if stack ended

Class PasTokenizerParallelStack
-------------------------------

Descendant of PasTokenizerStack, which uses thread(s) for parsing entire file. Before destroying it you mast call stop().

* __init__(lines, comments=True, qlong=1000): additonal param: size of internal queue buffer

Utils
-----

Helper functions to analyze token text.

* is_name(s): Check for valid identifier (can be reserved word too).
* is_comment(s): Check for valid comment.
* is_string(s): Check for string constant.


Author
=====
Artem Gavrilov (@Artem3213212 at Github)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pascal-tokenizer, version 1.0.4
Filename, size File type Python version Upload date Hashes
Filename, size pascal_tokenizer-1.0.4.tar.gz (3.5 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page