LLMVision
Visualize how LLMs tokenize text.
from llmvision import tokenize_and_visualize, GPT4Tokenizer
text = "Hello world! 👋🌍"
print(tokenize_and_visualize(text, GPT4Tokenizer()))
# Output: Hello│ world│!│<bytes:20f09f>│<bytes:91>│<bytes:8b>│<bytes:f09f>│<bytes:8c>│<bytes:8d>
Features
- Multiple tokenizers: GPT-2, GPT-4, byte-level, character-level
- Visual token boundaries
- Unicode/emoji handling
- Actual tokenization used by OpenAI models
Installation
pip install llmvision
Usage
llmvision "Hello world!"
llmvision "Hello world!" --tokenizer gpt4
llmvision "Hello world!" --indices
from llmvision import tokenize_and_visualize, GPT4Tokenizer
# Default tokenizer
print(tokenize_and_visualize("Hello world!"))
# Specific tokenizer
print(tokenize_and_visualize("Hello world!", GPT4Tokenizer()))
Examples
from llmvision import GPT4Tokenizer
tokenizer = GPT4Tokenizer()
tokens = tokenizer.tokenize("Hello world!")
print(tokens) # ['Hello', ' world', '!']
Tokenizers
SimpleTokenizer- word/punctuation/spaceWordTokenizer- whitespace-basedCharTokenizer- character-levelGraphemeTokenizer- Unicode grapheme clustersByteLevelTokenizer- UTF-8 bytesGPT2Tokenizer- GPT-2 (via tiktoken)GPT4Tokenizer- GPT-4 (via tiktoken)SubwordTokenizer- basic subword splitting
Token Costs
tokenizer = GPT4Tokenizer()
examples = [
"Hello world!", # 3 tokens
"Hello 世界!", # 5 tokens
"Hello 👋🌍!", # 8 tokens
"👨👩👧👦", # 18 tokens
]
for text in examples:
print(f"{text:15} → {len(tokenizer.tokenize(text))} tokens")
License
MIT
Metadata
Release files for llmvision 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| llmvision-0.1.1.tar.gz | 3.6 MB | Without provenance | Details for llmvision-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| llmvision-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for llmvision-0.1.1-py3-none-any.whl |
Total release size: 3.6 MB
Release files / llmvision-0.1.1.tar.gz
| Download URL | llmvision-0.1.1.tar.gz |
|---|---|
| Size | 3.6 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8eede57c89d8960e2a16cfb65d65a0841a697076f64ad2bbc6288f0a762a990b
|
|
BLAKE2b-256 checksum How to use checksums |
0b4bf829e91e1b8e0a5fd4fd416b69df6b0dbe710da62e964f87838621eee731
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.7.20
|
Release files / llmvision-0.1.1-py3-none-any.whl
| Download URL | llmvision-0.1.1-py3-none-any.whl |
|---|---|
| Size | 10.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
230d261fa1402e9c63be1cd01d16377c26c59fa1e32366ed1f95a6b921317a2a
|
|
BLAKE2b-256 checksum How to use checksums |
987565cc6dbb92346b6c400d19d0873253e0fd3b0e80d32dd658443f5045285c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.7.20
|
Security for llmvision 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.