Skip to main content

PyPI - Version PyPI - License Ruff Checked with mypy pre-commit Python versions CI/CD readthedocs

textmate-grammar-python

A Python port of the vscode-textmate grammar tokenization engine. It loads TextMate grammars and tokenizes source one line at a time, carrying the rule stack between lines.

The textmate_grammar.engine modules port the rule model, scope matcher, grammar loading and registry, and stateful tokenization loop. A differential suite compares token boundaries and scope paths against the real vscode-textmate 9.3.2 Node oracle across a 50-case MATLAB, Markdown, Python, and TypeScript corpus.

Usage

Requires Python 3.10 or newer. Install with:

pip install textmate-grammar-python

The public API follows vscode-textmate's registry and line-tokenization model. A registry loads raw grammar data by scope name; pass each line's returned ruleStack into the next call:

from pathlib import Path

from textmate_grammar import INITIAL, Registry, parseRawGrammar

grammar_path = Path("MATLAB.tmLanguage")
raw_grammar = parseRawGrammar(grammar_path.read_text(), str(grammar_path))
registry = Registry({"loadGrammar": lambda scope: raw_grammar if scope == "source.matlab" else None})
grammar = registry.loadGrammar("source.matlab")
state = INITIAL
for line in Path("example.m").read_text().splitlines():
    result = grammar.tokenizeLine(line, state)
    state = result.ruleStack
    print(result.tokens)  # each token has startIndex, endIndex, and scopes

The bundled MATLAB and Markdown grammar helpers are also available from textmate_grammar.parsers.matlab and textmate_grammar.parsers.markdown.

Migrating from 0.6.x

This 0.7.0 release is breaking within the 0.x series. The old LanguageParser / ContentElement tree API (including parse_file, parse_string, and flatten) is removed without compatibility shims. Replace it with Registry, parseRawGrammar, Grammar.tokenizeLine, IToken tokens, and the ruleStack state API shown above.

Information

Metadata

Release files for textmate-grammar-python 0.7.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for textmate-grammar-python 0.7.0
File Size Uploaded Provenance Actions
textmate_grammar_python-0.7.0.tar.gz 265.4 kB Without provenance Details for textmate_grammar_python-0.7.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for textmate-grammar-python 0.7.0
File Interpreter ABI Platform Actions
textmate_grammar_python-0.7.0-py3-none-any.whl Python 3 none any Details for textmate_grammar_python-0.7.0-py3-none-any.whl

Total release size: 336.1 kB

Release files / textmate_grammar_python-0.7.0.tar.gz

Download URL textmate_grammar_python-0.7.0.tar.gz
Size 265.4 kB
Tags Source
SHA-256 checksum
How to use checksums
1ad62179506eda4cc5bc10b02ff59c9b9969932e8f753622f75ada23fe4dfcd2
BLAKE2b-256 checksum
How to use checksums
d9a8d74a395d11658af12656c184fe6caa42874b1f2b7744a350c3f62172fac7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / textmate_grammar_python-0.7.0-py3-none-any.whl

Download URL textmate_grammar_python-0.7.0-py3-none-any.whl
Size 70.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b760fd285e0e0bbf18b2ab55dac5d3d2aafad4c533283bf7c83f397712f86e1d
BLAKE2b-256 checksum
How to use checksums
e1da4a6cd3e357c66ab94230dc3c45220b2f009d8ad51e4abcf798cab744fdcb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for textmate-grammar-python 0.7.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page