swarmauri_measurement_tokencountestimator 0.11.0.dev1

Token-budget estimation measurement for Swarmauri using tiktoken encodings and prompt-length preflight checks.

Swarmauri Measurement Token Count Estimator

swarmauri_measurement_tokencountestimator is the Swarmauri token-budget measurement for estimating text length with OpenAI's tiktoken. It helps you approximate prompt size before sending text into LLM, embedding, or chat-completion workflows.

Why Use Swarmauri Measurement Token Count Estimator

• Estimate prompt size before dispatching expensive model calls.
• Compare token usage across multiple tiktoken encodings.
• Reuse a standard Swarmauri MeasurementBase component in observability, cost-control, and prompt-construction flows.
• Add quick token-budget checks to preprocessing or routing pipelines.

FAQ

What does this measurement return?
An integer token count, or None when the requested encoding is invalid.

What encoding does it use by default?
cl100k_base.

Can I use another tokenizer?
Yes. Pass a different encoding name supported by tiktoken.

Does it raise on unknown encodings?
No. The current implementation prints the error and returns None.

Features

• Token-count estimation backed by tiktoken.
• Configurable encoding name for different model families.
• Simple Swarmauri measurement component for cost and context-window checks.
• Useful for prompt budgeting, chunk sizing, and preflight validation.
• Supports Python 3.10, 3.11, 3.12, 3.13, and 3.14.

Installation

uv add swarmauri_measurement_tokencountestimator

pip install swarmauri_measurement_tokencountestimator

Usage

from swarmauri_measurement_tokencountestimator import TokenCountEstimatorMeasurement

measurement = TokenCountEstimatorMeasurement()
text = "Lorem ipsum odor amet, consectetuer adipiscing elit."
count = measurement.calculate(text)
print(count)

Examples

Compare encodings for the same text

from swarmauri_measurement_tokencountestimator import TokenCountEstimatorMeasurement

text = "Swarmauri agents coordinate over shared memory."
measurement = TokenCountEstimatorMeasurement()

for encoding in ["cl100k_base", "o200k_base", "p50k_base"]:
    print(encoding, measurement.calculate(text, encoding=encoding))

Preflight a prompt budget

from swarmauri_measurement_tokencountestimator import TokenCountEstimatorMeasurement

measurement = TokenCountEstimatorMeasurement()
prompt = "Summarize the following customer support conversation..."

token_count = measurement.calculate(prompt)
if token_count is not None and token_count > 4000:
    print("Prompt should be chunked before dispatch.")

Handle unsupported encodings

from swarmauri_measurement_tokencountestimator import TokenCountEstimatorMeasurement

measurement = TokenCountEstimatorMeasurement()
print(measurement.calculate("Hello", encoding="not-real"))

Related Packages

• swarmauri_measurement_mutualinformation
• swarmauri_metric_hamming
• swarmauri_measurement_tokencountestimator

Swarmauri Foundations

• swarmauri
• swarmauri_core
• swarmauri_base

More Documentation

• OpenAI tiktoken repository
• OpenAI cookbook token counting examples

Best Practices

• Pin tiktoken if you need stable counts across builds and CI runs.
• Measure fully assembled prompts, not partial templates, when enforcing hard token budgets.
• Normalize whitespace consistently because tokenizers are sensitive to exact byte sequences.
• Treat None as an invalid-encoding signal and route it through explicit fallback logic.

License

This project is licensed under the Apache-2.0 License.