Swarmauri Measurement Token Count Estimator
Token-count measurement plugin for Swarmauri pipelines. Uses OpenAI's tiktoken library to estimate how many tokens a piece of text will consume given a specific tokenizer (default cl100k_base).
Features
- Implements the
MeasurementBaseAPI to slot into Swarmauri observability flows. - Wraps
tiktokenencoders for quick token-count estimation ahead of LLM calls. - Configurable tokenizer name so you can match counts to different model families.
Prerequisites
- Python 3.10 or newer.
tiktokeninstalled (pulled in automatically as a dependency).- If you use custom encodings, ensure they are registered with
tiktokenbefore invoking the measurement.
Installation
# pip
pip install swarmauri_measurement_tokencountestimator
# poetry
poetry add swarmauri_measurement_tokencountestimator
# uv (pyproject-based projects)
uv add swarmauri_measurement_tokencountestimator
Quickstart
from swarmauri_measurement_tokencountestimator import TokenCountEstimatorMeasurement
measurement = TokenCountEstimatorMeasurement()
text = "Lorem ipsum odor amet, consectetuer adipiscing elit."
count = measurement.calculate(text)
print(f"Tokens (cl100k_base): {count}")
Switching Encodings
from swarmauri_measurement_tokencountestimator import TokenCountEstimatorMeasurement
text = "Swarmauri agents coordinate over shared memory"
measurement = TokenCountEstimatorMeasurement()
for encoding in ["cl100k_base", "o200k_base", "p50k_base"]:
print(encoding, measurement.calculate(text, encoding=encoding))
Use this to check token budgets across different model families before dispatching a request.
Handling Unknown Encodings
from swarmauri_measurement_tokencountestimator import TokenCountEstimatorMeasurement
measurement = TokenCountEstimatorMeasurement()
invalid_count = measurement.calculate("Hello", encoding="not-real")
print(invalid_count) # Returns None and prints an error message
Wrap the call if you prefer structured error handling:
try:
count = measurement.calculate("Hello", encoding="not-real")
if count is None:
raise ValueError("Unsupported encoding")
except ValueError:
# fallback logic
pass
Tips
- Token counts can change as tokenizers evolve; pin
tiktokento a known version for stable measurements. - Normalize whitespace if your prompt assembly adds or strips spaces—tokenizers are sensitive to exact byte sequences.
- For batch estimation, combine this measurement with Pandas or list comprehensions to preprocess entire prompt sets before sending them to an LLM.
Want to help?
If you want to contribute to swarmauri-sdk, read up on our guidelines for contributing that will help you get started.
1 maintainer
Metadata
1 maintainer
Release files for swarmauri_measurement_tokencountestimator 0.9.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| swarmauri_measurement_tokencountestimator-0.9.3.tar.gz | 7.5 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| swarmauri_measurement_tokencountestimator-0.9.3-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 16.1 kB
Release files / swarmauri_measurement_tokencountestimator-0.9.3.tar.gz
| Download URL | swarmauri_measurement_tokencountestimator-0.9.3.tar.gz |
|---|---|
| Size | 7.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fd67d20cf8b873e27ed00377998f5276a1fc3e369a5d5a65de4acce30013486e
|
|
BLAKE2b-256 checksum How to use checksums |
ddd54757279bf3e72d74896ef63be23965a61363e583253c9ac4b57ae9143c2d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.0 {"installer":{"name":"uv","version":"0.11.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / swarmauri_measurement_tokencountestimator-0.9.3-py3-none-any.whl
| Download URL | swarmauri_measurement_tokencountestimator-0.9.3-py3-none-any.whl |
|---|---|
| Size | 8.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9ddaa80171389e1c03fc641ca90cfabf3895f0332de354681e34703891c2682a
|
|
BLAKE2b-256 checksum How to use checksums |
e69fc01e480e6d625e0cebf6c4d614aafdb8c7c5fc573a4c17bb075e4006ebb2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.0 {"installer":{"name":"uv","version":"0.11.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
