Skip to main content

PyPI - Version PyPI - License Ruff Checked with mypy pre-commit Python versions CI/CD readthedocs

textmate-grammar-python

A Python port of the vscode-textmate grammar tokenization engine. It loads TextMate grammars and tokenizes source one line at a time, carrying the rule stack between lines.

The textmate_grammar.engine modules port the rule model, scope matcher, grammar loading and registry, and stateful tokenization loop. A differential suite compares token boundaries and scope paths against the real vscode-textmate 9.3.2 Node oracle across a 50-case MATLAB, Markdown, Python, and TypeScript corpus.

Usage

Requires Python 3.10 or newer. Install with:

pip install textmate-grammar-python

The public API follows vscode-textmate's registry and line-tokenization model. A registry loads raw grammar data by scope name; pass each line's returned ruleStack into the next call:

from pathlib import Path

from textmate_grammar import INITIAL, Registry, parseRawGrammar

grammar_path = Path("MATLAB.tmLanguage")
raw_grammar = parseRawGrammar(grammar_path.read_text(), str(grammar_path))
registry = Registry({"loadGrammar": lambda scope: raw_grammar if scope == "source.matlab" else None})
grammar = registry.loadGrammar("source.matlab")
state = INITIAL
for line in Path("example.m").read_text().splitlines():
    result = grammar.tokenizeLine(line, state)
    state = result.ruleStack
    print(result.tokens)  # each token has startIndex, endIndex, and scopes

The bundled MATLAB and Markdown grammar helpers are also available from textmate_grammar.parsers.matlab and textmate_grammar.parsers.markdown.

Migrating from 0.6.x

This 1.0.0 release is the first stable release; it breaks the old 0.x element-tree API. The old LanguageParser / ContentElement tree API (including parse_file, parse_string, and flatten) is removed without compatibility shims. Replace it with Registry, parseRawGrammar, Grammar.tokenizeLine, IToken tokens, and the ruleStack state API shown above.

Information

Metadata

Release files for textmate-grammar-python 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for textmate-grammar-python 1.0.0
File Size Uploaded Provenance Actions
textmate_grammar_python-1.0.0.tar.gz 1.2 MB Without provenance Details for textmate_grammar_python-1.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for textmate-grammar-python 1.0.0
File Interpreter ABI Platform Actions
textmate_grammar_python-1.0.0-py3-none-any.whl Python 3 none any Details for textmate_grammar_python-1.0.0-py3-none-any.whl

Total release size: 1.2 MB

Release files / textmate_grammar_python-1.0.0.tar.gz

Download URL textmate_grammar_python-1.0.0.tar.gz
Size 1.2 MB
Tags Source
SHA-256 checksum
How to use checksums
a55e921acbd602d2c28c4328cc2b23bdebe8db04c7595bd01d2aa7ca532fd5ef
BLAKE2b-256 checksum
How to use checksums
e29bb60df44094efc230c1b58d3ab7961acc5008d51af3a897b04ad680d4c3f9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / textmate_grammar_python-1.0.0-py3-none-any.whl

Download URL textmate_grammar_python-1.0.0-py3-none-any.whl
Size 70.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
e57acefe622c53d79a9b856cbe155694e2dc92e8e780f61c2b601189c253feaa
BLAKE2b-256 checksum
How to use checksums
65fbaa720cd68e72e160caf984dd668420cb9fdd56da37c23f9cf9e315ceeed5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for textmate-grammar-python 1.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page