A code-optimized tokenizer for better understanding and processing of Python source code
Project description
CodeWeave
A code-optimized tokenizer designed specifically for source code, offering better compression and understanding of code structures compared to general-purpose tokenizers.
Installation
pip install codeweave
Quick Start
from codeweave import CodeTokenizer
tokenizer = CodeTokenizer()
code = """
def hello_world():
print("Hello, World!")
"""
tokens = tokenizer.tokenize(code)
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
codeweave-0.1.0.tar.gz
(2.8 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file codeweave-0.1.0.tar.gz.
File metadata
- Download URL: codeweave-0.1.0.tar.gz
- Upload date:
- Size: 2.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.12.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
618a5717481b896b18c9ba6fb092d772a52744f1d955da8e1e4a5834c34bc38b
|
|
| MD5 |
2d75b6fa1a69687f122a40af284b9d7f
|
|
| BLAKE2b-256 |
1cd2a462c689e2f44ccc0011ba67910c74e6be86ff90dfbb4bba82e446f41930
|
File details
Details for the file codeweave-0.1.0-py3-none-any.whl.
File metadata
- Download URL: codeweave-0.1.0-py3-none-any.whl
- Upload date:
- Size: 3.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.12.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
69d39c8202a0bedb400edd6d1a77cedca7430d75344531b2a2957c5135ff6839
|
|
| MD5 |
029af0aca45956fa69df9dad54c54c67
|
|
| BLAKE2b-256 |
e185b6d2934db1c5c55fb64da7d01d8d463d9f91892605bdfeaa5850d615178a
|
