swarmauri_measurement_tokencountestimator 0.9.3.dev24

This repository includes an example of a First Class Swarmauri Example.

---

Swarmauri Measurement Token Count Estimator

Token-count measurement plugin for Swarmauri pipelines. Uses OpenAI's tiktoken library to estimate how many tokens a piece of text will consume given a specific tokenizer (default cl100k_base).

Features

• Implements the MeasurementBase API to slot into Swarmauri observability flows.
• Wraps tiktoken encoders for quick token-count estimation ahead of LLM calls.
• Configurable tokenizer name so you can match counts to different model families.

Prerequisites

• Python 3.10 or newer.
• tiktoken installed (pulled in automatically as a dependency).
• If you use custom encodings, ensure they are registered with tiktoken before invoking the measurement.

Installation

# pip
pip install swarmauri_measurement_tokencountestimator

# poetry
poetry add swarmauri_measurement_tokencountestimator

# uv (pyproject-based projects)
uv add swarmauri_measurement_tokencountestimator

Quickstart

from swarmauri_measurement_tokencountestimator import TokenCountEstimatorMeasurement

measurement = TokenCountEstimatorMeasurement()
text = "Lorem ipsum odor amet, consectetuer adipiscing elit."
count = measurement.calculate(text)
print(f"Tokens (cl100k_base): {count}")

Switching Encodings

from swarmauri_measurement_tokencountestimator import TokenCountEstimatorMeasurement

text = "Swarmauri agents coordinate over shared memory"
measurement = TokenCountEstimatorMeasurement()

for encoding in ["cl100k_base", "o200k_base", "p50k_base"]:
    print(encoding, measurement.calculate(text, encoding=encoding))

Use this to check token budgets across different model families before dispatching a request.

Handling Unknown Encodings

from swarmauri_measurement_tokencountestimator import TokenCountEstimatorMeasurement

measurement = TokenCountEstimatorMeasurement()
invalid_count = measurement.calculate("Hello", encoding="not-real")
print(invalid_count)  # Returns None and prints an error message

Wrap the call if you prefer structured error handling:

try:
    count = measurement.calculate("Hello", encoding="not-real")
    if count is None:
        raise ValueError("Unsupported encoding")
except ValueError:
    # fallback logic
    pass

Tips

• Token counts can change as tokenizers evolve; pin tiktoken to a known version for stable measurements.
• Normalize whitespace if your prompt assembly adds or strips spaces—tokenizers are sensitive to exact byte sequences.
• For batch estimation, combine this measurement with Pandas or list comprehensions to preprocess entire prompt sets before sending them to an LLM.

Want to help?

If you want to contribute to swarmauri-sdk, read up on our guidelines for contributing that will help you get started.