Skip to main content

PyPI - Version PyPI - License Ruff Checked with ty prek Python versions CI/CD Documentation

textmate-grammar-python

A Python port of the vscode-textmate grammar tokenization engine. It loads TextMate grammars and tokenizes source one line at a time, carrying the rule stack between lines.

The textmate_grammar.engine modules port the rule model, scope matcher, grammar loading and registry, and stateful tokenization loop. A differential suite compares token boundaries and scope paths against the real vscode-textmate 9.3.2 Node oracle across a 50-case MATLAB, Markdown, Python, and TypeScript corpus.

Usage

Supports Python 3.10 through 3.15. Add it to a project with uv:

uv add textmate-grammar-python

The public API follows vscode-textmate's registry and line-tokenization model. A registry loads raw grammar data by scope name; pass each line's returned ruleStack into the next call:

from pathlib import Path

from textmate_grammar import INITIAL, Registry, parseRawGrammar

grammar_path = Path("MATLAB.tmLanguage")
raw_grammar = parseRawGrammar(grammar_path.read_text(), str(grammar_path))
registry = Registry({"loadGrammar": lambda scope: raw_grammar if scope == "source.matlab" else None})
grammar = registry.loadGrammar("source.matlab")
state = INITIAL
for line in Path("example.m").read_text().splitlines():
    result = grammar.tokenizeLine(line, state)
    state = result.ruleStack
    print(result.tokens)  # each token has startIndex, endIndex, and scopes

The bundled MATLAB and Markdown grammar helpers are also available from textmate_grammar.parsers.matlab and textmate_grammar.parsers.markdown.

Migrating from 0.6.x

This 1.0.0 release is the first stable release; it breaks the old 0.x element-tree API. The old LanguageParser / ContentElement tree API (including parse_file, parse_string, and flatten) is removed without compatibility shims. Replace it with Registry, parseRawGrammar, Grammar.tokenizeLine, IToken tokens, and the ruleStack state API shown above.

Information

Metadata

Release files for textmate-grammar-python 1.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for textmate-grammar-python 1.1.0
File Size Uploaded Provenance Actions
textmate_grammar_python-1.1.0.tar.gz 228.7 kB Without provenance Details for textmate_grammar_python-1.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for textmate-grammar-python 1.1.0
File Interpreter ABI Platform Actions
textmate_grammar_python-1.1.0-py3-none-any.whl Python 3 none any Details for textmate_grammar_python-1.1.0-py3-none-any.whl

Total release size: 299.5 kB

Release files / textmate_grammar_python-1.1.0.tar.gz

Download URL textmate_grammar_python-1.1.0.tar.gz
Size 228.7 kB
Tags Source
SHA-256 checksum
How to use checksums
138470aaaee59c57b0d36143a2931539fda5c9c6dbd53c8cf2eb65ea51071c73
BLAKE2b-256 checksum
How to use checksums
25bd97efef58224681b48c2b01ea6c5c1340bceee0bc86f2ce686b8d771a2120
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / textmate_grammar_python-1.1.0-py3-none-any.whl

Download URL textmate_grammar_python-1.1.0-py3-none-any.whl
Size 70.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
14665003e6ab118eae9e7d2232c1ddc301ac5902e6898c0172f924675f80ac40
BLAKE2b-256 checksum
How to use checksums
7514b44272d9108f2f0f66b620aa838a38e55ea32bd75de661061d979d9acdd8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for textmate-grammar-python 1.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page