AI21 Labs Tokenizer
A SentencePiece based tokenizer for production uses with AI21's models
Prerequisites
- If you wish to use the tokenizers for
Jamba MiniorJamba Large, you will need to request access to the relevant model's HuggingFace repo:
Installation
pip
pip install ai21-tokenizer
poetry
poetry add ai21-tokenizer
Usage
Basic Usage
from ai21_tokenizer import Tokenizer
# Create tokenizer (defaults to Jamba Mini)
tokenizer = Tokenizer.get_tokenizer()
# Encode text to token IDs
text = "Hello, world!"
encoded = tokenizer.encode(text)
print(f"Encoded: {encoded}")
# Decode token IDs back to text
decoded = tokenizer.decode(encoded)
print(f"Decoded: {decoded}")
Specific Tokenizer Selection
from ai21_tokenizer import Tokenizer, PreTrainedTokenizers
# Jamba Mini tokenizer
tokenizer = Tokenizer.get_tokenizer(PreTrainedTokenizers.JAMBA_MINI_TOKENIZER)
# Jamba Large tokenizer
tokenizer = Tokenizer.get_tokenizer(PreTrainedTokenizers.JAMBA_LARGE_TOKENIZER)
Async Usage
import asyncio
from ai21_tokenizer import Tokenizer
async def main():
tokenizer = await Tokenizer.get_async_tokenizer()
text = "Hello, world!"
encoded = await tokenizer.encode(text)
decoded = await tokenizer.decode(encoded)
print(f"Original: {text}")
print(f"Encoded: {encoded}")
print(f"Decoded: {decoded}")
asyncio.run(main())
Advanced Token Operations
# Convert between tokens and IDs
tokens = tokenizer.convert_ids_to_tokens(encoded)
print(f"Tokens: {tokens}")
ids = tokenizer.convert_tokens_to_ids(tokens)
print(f"IDs: {ids}")
Direct Class Usage
from ai21_tokenizer import SyncJambaTokenizer
# Using local model file
model_path = "/path/to/your/tokenizer.model"
tokenizer = SyncJambaTokenizer(model_path=model_path)
text = "Hello, world!"
encoded = tokenizer.encode(text)
decoded = tokenizer.decode(encoded)
Async Direct Class Usage
from ai21_tokenizer import AsyncJambaTokenizer
async def main():
model_path = "/path/to/your/tokenizer.model"
tokenizer = await AsyncJambaTokenizer.create(model_path=model_path)
text = "Hello, world!"
encoded = await tokenizer.encode(text)
decoded = await tokenizer.decode(encoded)
asyncio.run(main())
For more examples, please see our examples folder.
Metadata
Release files for ai21-tokenizer 1.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ai21_tokenizer-1.1.0.tar.gz | 2.6 MB | Without provenance | Details for ai21_tokenizer-1.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ai21_tokenizer-1.1.0-py3-none-any.whl | Python 3 | none | any | Details for ai21_tokenizer-1.1.0-py3-none-any.whl |
Total release size: 5.3 MB
Release files / ai21_tokenizer-1.1.0.tar.gz
| Download URL | ai21_tokenizer-1.1.0.tar.gz |
|---|---|
| Size | 2.6 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
aa511dc7d9717176fe37b24868cef5f8a568c4d3ab6ed336da69da10d7771c92
|
|
BLAKE2b-256 checksum How to use checksums |
0a96275e0000aaad03fe118afa964e669d69e0a8706dfbb079fde6837bf602c1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.9.23
|
Release files / ai21_tokenizer-1.1.0-py3-none-any.whl
| Download URL | ai21_tokenizer-1.1.0-py3-none-any.whl |
|---|---|
| Size | 2.7 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9402c8387c09a22b13995ea485b6d3ebe119c380bfaae733f2541da73457f7ed
|
|
BLAKE2b-256 checksum How to use checksums |
13fbd7ff8467745892a6fe65b0be7f6679654c9c6d1aba793bf6ccb8198120a5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.9.23
|
Security for ai21-tokenizer 1.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.