OpenAI Function Tokens Estimator
Estimate OpenAI token usage for chat completions, including functions, with this Python utility!
This package is based upon hmarr's openai-chat-tokens. As of right now (September 2023) there is no official documentation from openai on how to accurately predict the number of tokens from functions. This package solves that! Use it to get a very precise estimation of the token count for chat completions and better manage your OpenAI API usage.
Most often it is correct down to the token.
Installation
-
Install the Package via pip
pip install openai-function-tokens -
Import the Estimation Function
from openai_function_tokens import estimate_tokens
Usage
To use the estimator, call the estimate_tokens function:
estimate_tokens(messages, functions=None, function_call=None)
Pass in the messages, and optionally functions and function_call, to receive a precise token count.
Acknowledgments
Credit to hmarr for the original TypeScript tool. For a better understanding of token counting logic, check out his blog post.
Further Reading
How to call functions with chat models
How to use functions with a knowledge base
Counting tokens (only messages)
Contributing
Feedback, suggestions, and contributions are highly appreciated. Help make this tool even better!
1 maintainer
Metadata
1 maintainer
Release files for openai-function-tokens 0.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| openai-function-tokens-0.1.2.tar.gz | 3.8 kB | Without provenance | Details for openai-function-tokens-0.1.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| openai_function_tokens-0.1.2-py3-none-any.whl | Python 3 | none | any | Details for openai_function_tokens-0.1.2-py3-none-any.whl |
Total release size: 8.0 kB
Release files / openai-function-tokens-0.1.2.tar.gz
| Download URL | openai-function-tokens-0.1.2.tar.gz |
|---|---|
| Size | 3.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ab2d3479446f8a0d4e7aed6a08192bbeb71993fcf7aa8b6e1ef00a3fa313d1e5
|
|
BLAKE2b-256 checksum How to use checksums |
2f7ef9f0ef8f67d875c6146a89aeff497ebe633b48bbfbce07cfc97a6aa4541f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.11
|
Release files / openai_function_tokens-0.1.2-py3-none-any.whl
| Download URL | openai_function_tokens-0.1.2-py3-none-any.whl |
|---|---|
| Size | 4.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
346b9275db94f6f6b7b66d4ded06db0eb6321b3912c8ea2c9325eb697207833c
|
|
BLAKE2b-256 checksum How to use checksums |
429aaa58fcec9f30831329427527dcf8ba205c294ed6b112937a98bea01f4e67
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.11
|
Security for openai-function-tokens 0.1.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
