Skip to main content

build status pre-commit.ci status

tokenize-rt

The stdlib tokenize module does not properly roundtrip. This wrapper around the stdlib provides two additional tokens ESCAPED_NL and UNIMPORTANT_WS, and a Token data type. Use src_to_tokens and tokens_to_src to roundtrip.

This library is useful if you're writing a refactoring tool based on the python tokenization.

Installation

pip install tokenize-rt

Usage

datastructures

tokenize_rt.Offset(line=None, utf8_byte_offset=None)

A token offset, useful as a key when cross referencing the ast and the tokenized source.

tokenize_rt.Token(name, src, line=None, utf8_byte_offset=None)

Construct a token

  • name: one of the token names listed in token.tok_name or ESCAPED_NL or UNIMPORTANT_WS
  • src: token's source as text
  • line: the line number that this token appears on.
  • utf8_byte_offset: the utf8 byte offset that this token appears on in the line.

tokenize_rt.Token.offset

Retrieves an Offset for this token.

converting to and from Token representations

tokenize_rt.src_to_tokens(text: str) -> List[Token]

tokenize_rt.tokens_to_src(Iterable[Token]) -> str

additional tokens added by tokenize-rt

tokenize_rt.ESCAPED_NL

tokenize_rt.UNIMPORTANT_WS

helpers

tokenize_rt.NON_CODING_TOKENS

A frozenset containing tokens which may appear between others while not affecting control flow or code:

  • COMMENT
  • ESCAPED_NL
  • NL
  • UNIMPORTANT_WS

tokenize_rt.parse_string_literal(text: str) -> Tuple[str, str]

parse a string literal into its prefix and string content

>>> parse_string_literal('f"foo"')
('f', '"foo"')

tokenize_rt.reversed_enumerate(Sequence[Token]) -> Iterator[Tuple[int, Token]]

yields (index, token) pairs. Useful for rewriting source.

tokenize_rt.rfind_string_parts(Sequence[Token], i) -> Tuple[int, ...]

find the indices of the string parts of a (joined) string literal

  • i should start at the end of the string literal
  • returns () (an empty tuple) for things which are not string literals
>>> tokens = src_to_tokens('"foo" "bar".capitalize()')
>>> rfind_string_parts(tokens, 2)
(0, 2)
>>> tokens = src_to_tokens('("foo" "bar").capitalize()')
>>> rfind_string_parts(tokens, 4)
(1, 3)

Differences from tokenize

  • tokenize-rt adds ESCAPED_NL for a backslash-escaped newline "token"
  • tokenize-rt adds UNIMPORTANT_WS for whitespace (discarded in tokenize)
  • tokenize-rt normalizes string prefixes, even if they are not parsed -- for instance, this means you'll see Token('STRING', "f'foo'", ...) even in python 2.
  • tokenize-rt normalizes python 2 long literals (4l / 4L) and octal literals (0755) in python 3 (for easier rewriting of python 2 code while running python 3).

Sample usage

Metadata

Release files for tokenize-rt 5.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for tokenize-rt 5.1.0
File Size Uploaded Provenance Actions
tokenize_rt-5.1.0.tar.gz 5.3 kB Without provenance Details for tokenize_rt-5.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for tokenize-rt 5.1.0
File Interpreter ABI Platform Actions
tokenize_rt-5.1.0-py2.py3-none-any.whl Python 3, Python 2 none any Details for tokenize_rt-5.1.0-py2.py3-none-any.whl

Total release size: 11.0 kB

Release files / tokenize_rt-5.1.0.tar.gz

Download URL tokenize_rt-5.1.0.tar.gz
Size 5.3 kB
Tags Source
SHA-256 checksum
How to use checksums
08f0c2daa94c4052e53c2fcaa8e32585e6ae9bdfc800974092d031401694e002
BLAKE2b-256 checksum
How to use checksums
7307fe750a9c98b1449fc11f9f2893bff3cd093ddf966441bc126c8ffc75c6f0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.6

Release files / tokenize_rt-5.1.0-py2.py3-none-any.whl

Download URL tokenize_rt-5.1.0-py2.py3-none-any.whl
Size 5.8 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
9b7bb843e77dd6ed0be5564bfaaba200083911e0497841cd3e9235a6a9794d74
BLAKE2b-256 checksum
How to use checksums
33608fe577a3d0c7697df3701ad46300d967791d29a0a5225d9f11e84759ba26
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.6

Security for tokenize-rt 5.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page