tokencurator 0.1.0

Manage text content to fit specific token limits of machine learning models

TokenCurator

TokenCurator is a Python library designed to help developers manage and adjust text content to fit within specific token limits of machine learning models, particularly useful for working with models like OpenAI's GPT series. This library includes functionality to smartly truncate text content based on the encoding type and desired token constraints.

🌟 Features

• Get Tokenizer: Retrieve the appropriate tokenizer based on a specific encoding type.
• Truncate Text to Tokens: Limit the length of text to a specified number of tokens.
• Adjust Text: Adjust the text to fit within a specified total number of tokens, considering prompt and output token lengths.

🔧 Installation

To install TokenCurator, you will need Python 3.x installed on your system. It is recommended to install this package within a virtual environment.

pip install tokencurator  

🚀 Usage

Here's a quick example of how to use TokenCurator:

from tokencurator.openai import adjust_text

content = "Your very long input text here..."
prompt_token_length = 100  # Number of tokens for the prompt
output_token_length = 100  # Number of tokens reserved for the model's output
max_total_tokens = 2048  # Maximum tokens your model configuration allows

# Adjust the text to fit within the maximum token limit
adjusted_text = adjust_text(content, prompt_token_length, output_token_length, max_total_tokens, "gpt-3.5-turbo")
print(adjusted_text)

🧪 Testing

To run tests, you'll need to have pytest installed. You can run the tests to ensure everything is working as expected by navigating to the package directory and running:

pytest

🤖 Supported OpenAI Models

TokenCurator is designed to work with a variety of OpenAI models, providing robust support for managing tokenization constraints specific to each model. Below is a table listing the supported models and their respective encodings:

Model Category	Model Name	Encoding
Chat Models	GPT-4	cl100k_base
	GPT-3.5 Turbo	cl100k_base
	GPT-3.5	cl100k_base
	GPT-35 Turbo (Azure)	cl100k_base
Base Models	Davinci-002	cl100k_base
	Babbage-002	cl100k_base
Embedding Models	Text Embedding Ada-002	cl100k_base
	Text Embedding 3 Small	cl100k_base
	Text Embedding 3 Large	cl100k_base
Deprecated Models	Text Davinci-003	p50k_base
	Text Davinci-002	p50k_base
	Davinci	r50k_base
	Curie	r50k_base
	Babbage	r50k_base
	Ada	r50k_base
	Code Davinci-002	p50k_base
	Davinci Codex	p50k_base
	Text Davinci Edit-001	p50k_edit
Open Source Models	GPT-2	gpt2

This table ensures that TokenCurator can seamlessly interface with a wide range of models, helping you manage and adjust tokenization effectively for your specific use case.

📚 Additional Resources

For more tools, libraries, and tutorials, visit our official website: Grade Calculator.

License 📜

This project is licensed under the Apache 2 License - see the LICENSE file for details.