An interpreter for grammar files as defined by TextMate and used in VSCode, implemented in Python. TextMate grammars use the oniguruma dialect (https://github.com/kkos/oniguruma). Supports loading grammar files from JSON, PLIST, or YAML format.
Project description
textmate-grammar-python
An interpreter for grammar files as defined by TextMate and used in VSCode, implemented in Python. TextMate grammars use the oniguruma dialect (https://github.com/kkos/oniguruma). Supports loading grammar files from JSON, PLIST, or YAML format.
Usage
Install the module using pip install textmate-grammar-python
.
Before tokenization is possible, a LanguageParser
needs to be initialized using a loaded grammar.
from textmate_grammar.language import LanguageParser
from textmate_grammar.grammars import matlab
parser = LanguageParser(matlab.GRAMMAR)
After this, one can either choose to call parser.parsing_string
to parse a input string directly, or call parser.parse_file
with the path to the appropiate source file as the first argument, such as the the example example.py
.
The parsed element
object can be displayed directly by calling the print
method. By default the element is printed as an element tree in a dictionary format.
>>> element = parser.parse_string("value = num2str(10);")
>>> element.print()
{'token': 'source.matlab',
'children': [{'token': 'meta.assignment.variable.single.matlab',
'children': [{'token': 'variable.other.readwrite.matlab', 'content': 'value'}]},
{'token': 'keyword.operator.assignment.matlab', 'content': '='},
{'token': 'meta.function-call.parens.matlab',
'begin': [{'token': 'entity.name.function.matlab', 'content': 'num2str'},
{'token': 'punctuation.section.parens.begin.matlab', 'content': '('}],
'end': [{'token': 'punctuation.section.parens.end.matlab', 'content': ')'}],
'children': [{'token': 'constant.numeric.decimal.matlab', 'content': '10'}]},
{'token': 'punctuation.terminator.semicolon.matlab', 'content': ';'}]}
Alternatively, with the keyword argument flatten
the element is displayed as a list per unique token. Here the first item in the list is the starting position (line, column) of the unique tokenized element.
>>> element.print(flatten=True)
[[(0, 0), 'value', ['source.matlab', 'meta.assignment.variable.single.matlab', 'variable.other.readwrite.matlab']],
[(0, 5), ' ', ['source.matlab']],
[(0, 6), '=', ['source.matlab', 'keyword.operator.assignment.matlab']],
[(0, 7), ' ', ['source.matlab']],
[(0, 8), 'num2str', ['source.matlab', 'meta.function-call.parens.matlab', 'entity.name.function.matlab']],
[(0, 15), '(', ['source.matlab', 'meta.function-call.parens.matlab', 'punctuation.section.parens.begin.matlab']],
[(0, 16), '10', ['source.matlab', 'meta.function-call.parens.matlab', 'constant.numeric.decimal.matlab']],
[(0, 18), ')', ['source.matlab', 'meta.function-call.parens.matlab', 'punctuation.section.parens.end.matlab']],
[(0, 19), ';', ['source.matlab', 'punctuation.terminator.semicolon.matlab']]]
Supported Languages
TODO
- Implement Begin/While pattern, required for other grammars.
Sources
Development
Install the repository after cloning with poetry
> pip install poetry
> cd textmate-grammar-python
> poetry install
Setup nodejs dependencies in the regression testing folder.
> cd test/regression
> bash install.js
Run all tests in main test directory
> cd ..
> pytest
====================== test session starts =====================
platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0
rootdir: /home/watermarkhu/repositories/sphinx-matlab/test
plugins: icdiff-0.8
...
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for textmate_grammar_python-0.1.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 23052477bf6db03ef4d8fbb450f9319df15a5ad8c6e862e56aa1d93ff0350a55 |
|
MD5 | 9f3598afee631a282554b7df83f6c9dd |
|
BLAKE2b-256 | e64c6ed2883e1a651632891399f9a6c986853c49386f639d0e46edc27bea1c45 |
Hashes for textmate_grammar_python-0.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d967d8b07dd9e3b2b689ae7c6be532925b077ad5733e90673515b7b5e866c939 |
|
MD5 | 7392a596aa813cbb377a7d926e7c6ab2 |
|
BLAKE2b-256 | a8e853399e2b4af49d515981932739d44e4528b7d6e5e80a54e70c62ea85188f |