token-utils

Installation: pip install token-utils.

The purpose of token-utils is to simplify manipulations of tokens normally obtaind from Python's tokenize module. One of token-utils' features is that, unlike Python's version, the following is always guaranteed:

from token_utils import tokenize, untokenize

source = "Arbitrary Python code here"

assert source == untokenize(tokenize(source))

To get an idea of the simplicity of using token-utils, consider this example from Python's standard library which substitute Decimals for floats in a string of statements:

from tokenize import tokenize, untokenize, NUMBER, STRING, NAME, OP
from io import BytesIO

def decistmt(s):
    result = []
    g = tokenize(BytesIO(s.encode('utf-8')).readline)  # tokenize the string
    for toknum, tokval, _, _, _ in g:
        if toknum == NUMBER and '.' in tokval:  # replace NUMBER tokens
            result.extend([
                (NAME, 'Decimal'),
                (OP, '('),
                (STRING, repr(tokval)),
                (OP, ')')
            ])
        else:
            result.append((toknum, tokval))
    return untokenize(result).decode('utf-8')

Here's how you could achieve the same result with token-utils:

from token_utils import tokenize, untokenize

def decistmt(source):
    tokens = tokenize(source)
    for token in tokens:
        if token.is_float():
            token.string = f"Decimal('{token.string}')"
    return untokenize(tokens)

See the documentation for more information.

Release files for token-utils 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for token-utils 0.2.0
File	Size	Uploaded
token_utils-0.2.0.tar.gz	9.4 MB	Sep 25, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for token-utils 0.2.0
File	Interpreter	ABI	Platform	Reset
token_utils-0.2.0-py3-none-any.whl 15.9 kB Sep 25, 2026	Python 3	none	any	Details

Total release size: 9.4 MB

Release files / token_utils-0.2.0.tar.gz

Download URL	token_utils-0.2.0.tar.gz
Size	9.4 MB
Tags	Source
SHA-256 checksum How to use checksums	`fedc62b2e57c1f803e6e1f53b4d775d3e7b07cf6c9e0c4b56012fb999f327491`
BLAKE2b-256 checksum How to use checksums	`5ac1f25c7c4fc1fa7ee40313e034a537dc4d88af6cb6c3c6d211d4558b59fb56`
Upload date	Sep 25, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.9.13`

Release files / token_utils-0.2.0-py3-none-any.whl

Download URL	token_utils-0.2.0-py3-none-any.whl
Size	15.9 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`1f2d9fee2355d23fe861a6ae31e28094e8755effe11311a8d1da88b644dc690d`
BLAKE2b-256 checksum How to use checksums	`98be5e9e397e596f7315a745db9b9af7f972c4017b4dd31f44b36f30cc0b9d76`
Upload date	Sep 25, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.2.0 CPython/3.9.13`