Skip to main content

Tokenize code snippets.

Project description

Tokenize-All

Tokenize blocks of code in Python. Used by manim-code-blocks to syntax highlight blocks of code.

Example Usage

from tokenize_all import Java

tokens = Java.tokenize(
    """
    public class Main {
        public static void main(String[] args) {
            System.out.println("Hello world!");
        }
    }
    """
)

for token in tokens: print(token)

Output:

>> Token[ type = keyword, value = public, start = 4, end = 10 ]
>> Token[ type = keyword, value = class, start = 11, end = 16 ]
>> Token[ type = class name, value = Main, start = 17, end = 21 ]
>> Token[ type = left brace, value = {, start = 22, end = 23 ]
>> Token[ type = keyword, value = public, start = 31, end = 37 ]
>> Token[ type = keyword, value = static, start = 38, end = 44 ]
>> Token[ type = keyword, value = void, start = 45, end = 49 ]
>> Token[ type = function, value = main, start = 50, end = 54 ]
>> Token[ type = left parentheses, value = (, start = 54, end = >> 55 ]
>> Token[ type = class name, value = String, start = 55, end = >> 61 ]
>> Token[ type = left bracket, value = [, start = 61, end = 62 ]
>> Token[ type = right bracket, value = ], start = 62, end = 63 ]
>> Token[ type = identifier, value = args, start = 64, end = 68 ]
>> Token[ type = right parentheses, value = ), start = 68, end = >> 69 ]
>> Token[ type = left brace, value = {, start = 70, end = 71 ]
>> Token[ type = class name, value = System, start = 83, end = >> 89 ]
>> Token[ type = dot, value = ., start = 89, end = 90 ]
>> Token[ type = identifier, value = out, start = 90, end = 93 ]
>> Token[ type = dot, value = ., start = 93, end = 94 ]
>> Token[ type = function, value = println, start = 94, end = >> 101 ]
>> Token[ type = left parentheses, value = (, start = 101, end = >> 102 ]
>> Token[ type = string, value = "Hello world!", start = 102, >> end = 116 ]
>> Token[ type = right parentheses, value = ), start = 116, end >> = 117 ]
>> Token[ type = semicolon, value = ;, start = 117, end = 118 ]
>> Token[ type = right brace, value = }, start = 126, end = 127 ]
>> Token[ type = right brace, value = }, start = 131, end = 132 ]

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tokenize_all-1.0.4.tar.gz (6.7 kB view details)

Uploaded Source

Built Distribution

tokenize_all-1.0.4-py3-none-any.whl (7.5 kB view details)

Uploaded Python 3

File details

Details for the file tokenize_all-1.0.4.tar.gz.

File metadata

  • Download URL: tokenize_all-1.0.4.tar.gz
  • Upload date:
  • Size: 6.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.4

File hashes

Hashes for tokenize_all-1.0.4.tar.gz
Algorithm Hash digest
SHA256 5ca654c96128b8e503b6a2b273c44b23f842a9060e17c9252dc4d004aac6f618
MD5 6b8b9fc7ed17667cdd2d63fdfc1efdf8
BLAKE2b-256 6829dfb619f793ebf88cb9ba75c8700087d13ba89b68b92d7742143d88bcb2f2

See more details on using hashes here.

File details

Details for the file tokenize_all-1.0.4-py3-none-any.whl.

File metadata

File hashes

Hashes for tokenize_all-1.0.4-py3-none-any.whl
Algorithm Hash digest
SHA256 28a7700875f1c001f63e3e4eb008cff900e2201409ddb4eb6f680e3ace36ac8a
MD5 4e439150ef6d9a9885e22bad0662e3fb
BLAKE2b-256 970ebd29b5ef2a3485b7ec265c9cc759cfa8be6c69af0538cff4d7ca9461d669

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page