token-utils
Installation: pip install token-utils.
The purpose of token-utils is to simplify manipulations of tokens normally obtaind from Python's tokenize module. One of token-utils' features is that, unlike Python's version, the following is always guaranteed:
from token_utils import tokenize, untokenize
source = "Arbitrary Python code here"
assert source == untokenize(tokenize(source))
To get an idea of the simplicity of using token-utils, consider this example from Python's standard library which substitute Decimals for floats in a string of statements:
from tokenize import tokenize, untokenize, NUMBER, STRING, NAME, OP
from io import BytesIO
def decistmt(s):
result = []
g = tokenize(BytesIO(s.encode('utf-8')).readline) # tokenize the string
for toknum, tokval, _, _, _ in g:
if toknum == NUMBER and '.' in tokval: # replace NUMBER tokens
result.extend([
(NAME, 'Decimal'),
(OP, '('),
(STRING, repr(tokval)),
(OP, ')')
])
else:
result.append((toknum, tokval))
return untokenize(result).decode('utf-8')
Here's how you could achieve the same result with token-utils:
from token_utils import tokenize, untokenize
def decistmt(source):
tokens = tokenize(source)
for token in tokens:
if token.is_float():
token.string = f"Decimal('{token.string}')"
return untokenize(tokens)
See the documentation for more information.
1 maintainer
Metadata
1 maintainer
Release files for token-utils 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| token_utils-0.3.0.tar.gz | 7.8 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| token_utils-0.3.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 7.8 MB
Release files / token_utils-0.3.0.tar.gz
| Download URL | token_utils-0.3.0.tar.gz |
|---|---|
| Size | 7.8 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fda7725aa7775bc2f11acf0267c4bb75118cbae701b87c7dcbe4593f09ce8596
|
|
BLAKE2b-256 checksum How to use checksums |
028dff3994b54f2ec139bf693b1694592d404ff1a7954d2e724309cc07a64e42
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.11
|
Release files / token_utils-0.3.0-py3-none-any.whl
| Download URL | token_utils-0.3.0-py3-none-any.whl |
|---|---|
| Size | 19.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8ac3eba15f0f4fdc661d92b1de957b7e7f67539f09fd8c55f4b170882d04db96
|
|
BLAKE2b-256 checksum How to use checksums |
8cc3aa7b7947290020fc2738af321a9c07bcf4cb3c1b7204b926838af8db7c02
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.11
|
