textmate-grammar-python
A Python port of the vscode-textmate grammar tokenization engine. It loads TextMate grammars and tokenizes source one line at a time, carrying the rule stack between lines.
The textmate_grammar.engine modules port the rule model, scope matcher, grammar loading and registry, and stateful tokenization loop. A differential suite compares token boundaries and scope paths against the real vscode-textmate 9.3.2 Node oracle across a 50-case MATLAB, Markdown, Python, and TypeScript corpus.
Usage
Requires Python 3.10 or newer. Install with:
pip install textmate-grammar-python
The public API follows vscode-textmate's registry and line-tokenization model. A registry loads raw grammar data by scope name; pass each line's returned ruleStack into the next call:
from pathlib import Path
from textmate_grammar import INITIAL, Registry, parseRawGrammar
grammar_path = Path("MATLAB.tmLanguage")
raw_grammar = parseRawGrammar(grammar_path.read_text(), str(grammar_path))
registry = Registry({"loadGrammar": lambda scope: raw_grammar if scope == "source.matlab" else None})
grammar = registry.loadGrammar("source.matlab")
state = INITIAL
for line in Path("example.m").read_text().splitlines():
result = grammar.tokenizeLine(line, state)
state = result.ruleStack
print(result.tokens) # each token has startIndex, endIndex, and scopes
The bundled MATLAB and Markdown grammar helpers are also available from textmate_grammar.parsers.matlab and textmate_grammar.parsers.markdown.
Migrating from 0.6.x
This 0.7.0 release is breaking within the 0.x series. The old LanguageParser / ContentElement tree API (including parse_file, parse_string, and flatten) is removed without compatibility shims. Replace it with Registry, parseRawGrammar, Grammar.tokenizeLine, IToken tokens, and the ruleStack state API shown above.
Information
- For further information, please checkout the documentation.
- To setup an environment for development, see CONTRIBUTING.md
Metadata
Release files for textmate-grammar-python 0.7.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| textmate_grammar_python-0.7.0.tar.gz | 265.4 kB | Without provenance | Details for textmate_grammar_python-0.7.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| textmate_grammar_python-0.7.0-py3-none-any.whl | Python 3 | none | any | Details for textmate_grammar_python-0.7.0-py3-none-any.whl |
Total release size: 336.1 kB
Release files / textmate_grammar_python-0.7.0.tar.gz
| Download URL | textmate_grammar_python-0.7.0.tar.gz |
|---|---|
| Size | 265.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1ad62179506eda4cc5bc10b02ff59c9b9969932e8f753622f75ada23fe4dfcd2
|
|
BLAKE2b-256 checksum How to use checksums |
d9a8d74a395d11658af12656c184fe6caa42874b1f2b7744a350c3f62172fac7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / textmate_grammar_python-0.7.0-py3-none-any.whl
| Download URL | textmate_grammar_python-0.7.0-py3-none-any.whl |
|---|---|
| Size | 70.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b760fd285e0e0bbf18b2ab55dac5d3d2aafad4c533283bf7c83f397712f86e1d
|
|
BLAKE2b-256 checksum How to use checksums |
e1da4a6cd3e357c66ab94230dc3c45220b2f009d8ad51e4abcf798cab744fdcb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for textmate-grammar-python 0.7.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.