Skip to main content

Tokenize All Code

This is a clone of the original code that was found here. That repo has since disappeared, so I have cloned it here for archival and development purposes.

Tokenize blocks of code in Python. Used by manim-code-blocks to syntax highlight blocks of code.

Example Usage

from tokenize_all import Java

tokens = Java.tokenize(
    """
    public class Main {
        public static void main(String[] args) {
            System.out.println("Hello world!");
        }
    }
    """
)

for token in tokens: print(token)

Output:

>> Token[ type = keyword, value = public, start = 4, end = 10 ]
>> Token[ type = keyword, value = class, start = 11, end = 16 ]
>> Token[ type = class name, value = Main, start = 17, end = 21 ]
>> Token[ type = left brace, value = {, start = 22, end = 23 ]
>> Token[ type = keyword, value = public, start = 31, end = 37 ]
>> Token[ type = keyword, value = static, start = 38, end = 44 ]
>> Token[ type = keyword, value = void, start = 45, end = 49 ]
>> Token[ type = function, value = main, start = 50, end = 54 ]
>> Token[ type = left parentheses, value = (, start = 54, end = >> 55 ]
>> Token[ type = class name, value = String, start = 55, end = >> 61 ]
>> Token[ type = left bracket, value = [, start = 61, end = 62 ]
>> Token[ type = right bracket, value = ], start = 62, end = 63 ]
>> Token[ type = identifier, value = args, start = 64, end = 68 ]
>> Token[ type = right parentheses, value = ), start = 68, end = >> 69 ]
>> Token[ type = left brace, value = {, start = 70, end = 71 ]
>> Token[ type = class name, value = System, start = 83, end = >> 89 ]
>> Token[ type = dot, value = ., start = 89, end = 90 ]
>> Token[ type = identifier, value = out, start = 90, end = 93 ]
>> Token[ type = dot, value = ., start = 93, end = 94 ]
>> Token[ type = function, value = println, start = 94, end = >> 101 ]
>> Token[ type = left parentheses, value = (, start = 101, end = >> 102 ]
>> Token[ type = string, value = "Hello world!", start = 102, >> end = 116 ]
>> Token[ type = right parentheses, value = ), start = 116, end >> = 117 ]
>> Token[ type = semicolon, value = ;, start = 117, end = 118 ]
>> Token[ type = right brace, value = }, start = 126, end = 127 ]
>> Token[ type = right brace, value = }, start = 131, end = 132 ]

Building and Publishing

Requires Python 3.9 or later.

pip install twine build
python -m build
twine upload dist/*

Metadata

Release files for tokenize-all-code 1.0.18

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for tokenize-all-code 1.0.18
File Size Uploaded Provenance Actions
tokenize_all_code-1.0.18.tar.gz 7.8 kB Without provenance Details for tokenize_all_code-1.0.18.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for tokenize-all-code 1.0.18
File Interpreter ABI Platform Actions
tokenize_all_code-1.0.18-py3-none-any.whl Python 3 none any Details for tokenize_all_code-1.0.18-py3-none-any.whl

Total release size: 15.3 kB

Release files / tokenize_all_code-1.0.18.tar.gz

Download URL tokenize_all_code-1.0.18.tar.gz
Size 7.8 kB
Tags Source
SHA-256 checksum
How to use checksums
1af1913a8e5cdb3d7d5b2e03e3e77947b86ae2396a9d08ddeb851edac8ef5f6d
BLAKE2b-256 checksum
How to use checksums
fd36e1dea71eded238a6c52777432331ae80cde22789376878e35d2d9b0c8f51
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.4

Release files / tokenize_all_code-1.0.18-py3-none-any.whl

Download URL tokenize_all_code-1.0.18-py3-none-any.whl
Size 7.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
dcd05740d11db934c371a292f83f667cd5d5458006812f436a8592d3320ef1e3
BLAKE2b-256 checksum
How to use checksums
0c06b104e99ce5e76a8060f87022542253e15e55cdafe592396d1551eec14d8e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.10.4

Security for tokenize-all-code 1.0.18

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.18 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page