token-calibrator 1.0.1

Calibrated, self-learning token estimator for live context-size display

token-calibrator

Tokenizer-free token estimation for any LLM.

Learn the tokenizer instead of shipping the tokenizer.

Instead of shipping a tokenizer, token-calibrator learns how your model tokenizes text from real API usage. Estimates become more accurate over time, without vendor-specific code or tokenizer updates.

Works with GPT, Claude, Gemini, DeepSeek, Qwen, Llama, Mistral, and future models.

---

Why?

Most LLM applications only need an approximate token count to:

• Display live context usage
• Estimate API cost
• Warn before exceeding context limits
• Budget prompts while editing

The traditional solution is to bundle a tokenizer for every supported model. That works—but it comes with trade-offs.

Traditional tokenizer	token-calibrator
Bundled tokenizer	No tokenizer
Model-specific	Model-agnostic
Requires tokenizer updates	Learns automatically
Large vocabulary tables	Seven learned coefficients
Doesn't support unknown models	Works with future models

Instead of reproducing a tokenizer, token-calibrator learns the relationship between text and token counts directly from your provider's responses.

---

Features

• 🚀 Tokenizer-free
• 🤖 Works with any LLM
• 📈 Learns continuously from real API responses
• 🧠 Online ridge regression with configurable forgetting
• 💾 Tiny memory footprint (only seven learned coefficients)
• 📦 Serializable accumulator state
• ⚡ Streaming-friendly
• 🌍 Available for TypeScript, Rust, Python, and Go

---

How it works

             Input text
                  │
                  ▼
      Character classification
 Han / Latin / Digit / Hangul /
  Cyrillic / Emoji / Other
                  │
                  ▼
      Current per-bucket rates
                  │
                  ▼
      Estimated token count
                  │
                  ▼
     observe(real_tokens)
                  │
                  ▼
      Online ridge regression

Each observation adjusts the coefficients slightly, making future estimates better match the tokenizer actually used by your model.

---

Buckets

Characters are grouped into seven buckets to capture how different scripts tokenize:

Bucket	Count unit	Scripts / characters	Typical tokens/unit
Han	character	CJK ideographs (Chinese, Japanese Kanji)	~0.9–1.3
Latin	character	ASCII, Latin-1 Supplement, punctuation, spaces	~0.23–0.25
Digit	character	0–9	~0.4–1.3
Hangul	character	Korean Hangul syllables + Jamo	~0.8–1.5
Cyrillic	character	Cyrillic + Supplement	~0.3–0.5
Emoji	character	Emoticons, pictographs, flags, dingbats	~1.4–3.2
Other	UTF-8 byte	Everything else (kana, Arabic, Thai, control chars...)	~0.16–0.40

Every bucket is counted in characters except other, which is counted in UTF-8 bytes — scripts outside a tokenizer's merge vocabulary fall back to byte-level BPE (≈ 1 token/byte).

---

Languages

Language	Package	Install	Location
TypeScript	npm (@zlogic/token-calibrator)	npm install @zlogic/token-calibrator	ts/
Python	PyPI (token-calibrator)	pip install token-calibrator	python/
Rust	crates.io (token-calibrator)	cargo add token-calibrator	rust/
Go	go get github.com/zlogic-libs/token-calibrator/go	go get github.com/zlogic-libs/...	go/

---

Quick start

TypeScript

npm install @zlogic/token-calibrator

import { TokenCalibrator, TokenEstimator } from "@zlogic/token-calibrator";

// ── Train from real usage ──
const cal = new TokenCalibrator();
cal.observe(prompt, actualTokens);
const matrix = cal.toMatrix(); // save per model

// ── Estimate tokens by model name ──
// Uses built-in baseline (includes gpt-4o, deepseek-chat, llama-3.1-70b, ...)
const estimator = new TokenEstimator({ "my-model": matrix });
const tokens = estimator.estimate("gpt-4o", prompt);

Python

pip install token-calibrator

from token_calibrator import TokenCalibrator, TokenEstimator

# Train
cal = TokenCalibrator()
cal.observe(prompt, actual_tokens)
matrix = cal.to_matrix()  # save per model

# Estimate by model name — uses built-in baseline models
estimator = TokenEstimator({"my-model": matrix})
tokens = estimator.estimate("gpt-4o", prompt)

Rust

cargo add token-calibrator

use token_calibrator::{TokenCalibrator, TokenCalibratorOptions, TokenEstimator, TokenEstimatorOptions};
use std::collections::HashMap;

// Train
let mut cal = TokenCalibrator::new(TokenCalibratorOptions::default());
cal.observe(prompt, actual_tokens);
let matrix = cal.to_matrix();  // clone for persistence

// Estimate by model name — uses built-in baseline models
let mut matrices = HashMap::new();
matrices.insert("my-model".into(), matrix);
let estimator = TokenEstimator::new(Some(matrices), TokenEstimatorOptions::default());
let tokens = estimator.estimate("gpt-4o", prompt);

Go

go get github.com/zlogic-libs/token-calibrator/go

import calibrator "github.com/zlogic-libs/token-calibrator/go"

// Train
cal := calibrator.NewTokenCalibrator(calibrator.TokenCalibratorOptions{})
cal.Observe(prompt, actualTokens)
matrix := cal.ToMatrix()

// Estimate by model name — uses built-in baseline models
matrices := map[string]calibrator.TokenAccumulator{"my-model": matrix}
estimator := calibrator.NewTokenEstimator(matrices, calibrator.TokenEstimatorOptions{})
tokens := estimator.Estimate("gpt-4o", prompt)

---

Examples

Run a complete walkthrough in your language of choice. The demo has two modes:

• calibrate — feeds sample observations, shows learned rates, exports accumulator to calibrated-snapshot.json
• estimate — loads calibrated data (or uses built-in models) and estimates token counts for several text types

Language	Command (calibrate)	Command (estimate)
TypeScript	npx tsx examples/demo.ts calibrate	npx tsx examples/demo.ts estimate [file]
Python	python examples/demo.py calibrate	python examples/demo.py estimate [file]
Rust	cargo run --example demo calibrate	cargo run --example demo estimate [file]
Go	go run ./cmd/demo calibrate	go run ./cmd/demo estimate [file]

The calibrate output:

1. Shows each observation with its per-bucket classification
2. Prints the learned per-bucket rates
3. Shows estimated tokens for a few test strings
4. Exports the accumulator as trained-snapshot.json

The estimate output:

1. If a file argument is given, loads that JSON snapshot; otherwise uses built-in models only
2. Shows estimated tokens for several text types across different models

---

Use cases

Ideal for applications that need fast, lightweight token estimation:

• AI chat clients
• Agent frameworks
• IDE extensions
• Prompt editors
• Live context meters
• Cost estimation
• Token budgeting
• Streaming interfaces

---

Algorithm

The estimator models:

tokens ≈ Σ αᵢ × bucket_countᵢ

Characters are grouped into seven buckets:

• Han (CJK ideographs)
• Latin (ASCII + Latin-1 Supplement)
• Digit
• Hangul
• Cyrillic
• Emoji
• Other (measured in UTF-8 bytes)

Every bucket is a per-character rate, except other which is a per-byte rate. The coefficients are learned using online ridge regression by solving:

(XᵀX + λI)α = Xᵀy

Because there are only seven coefficients, the system solves a fixed 7×7 linear system using Gauss–Jordan elimination. Exponential forgetting (γ) is supported for streaming environments where tokenizer behavior may drift over time.

---

API reference

TokenCalibrator (writer / trainer)

Method	Description
TokenCalibrator(opts?)	Create a calibrator
.observe(input, real_tokens)	Feed one observation
.rates() → TokenRates	Current learned per-bucket rates
.estimate(input) → number	Estimate tokens using current rates
.to_matrix() → TokenAccumulator	Raw data-only accumulator (persist)

TokenEstimator (reader)

Method	Description
TokenEstimator(matrices?, opts?)	Create estimator from model matrices
.estimate(model_name, input) → number	Estimate tokens by model name
.rates(model_name) → TokenRates	Effective rates for a model
.has(model_name) → bool	Whether model has user-calibrated data

Pure functions

Function	Description
classify_token_buckets(input) → counts	Classify text into per-bucket counts
feature_vector(counts) → [f64; 7]	Ordered feature vector for regression
estimate_tokens(input, rates) → number	Pure token estimate
empty_accumulator() → TokenAccumulator	Fresh zero-initialised accumulator
accumulate(acc, input, real_tokens, forgetting?) → acc	Fold one observation (pure)
rates_from_accumulator(acc, priorStrength?, prior?) → ...	Solve ridge regression
is_valid_accumulator(acc) → bool	Structural validity check
solve_linear_system(a, b) → [f64; 7]	Gauss–Jordan elimination
derive_rates(matrices, ...) → {model: rates}	Batch derive rates from multiple accs

Constants

Constant	Type	Description
TOKEN_BUCKETS	string[]	Ordered bucket names
N_BUCKETS	number / usize / int	Number of buckets (7)
TOKEN_BUCKET_PRIORS	TokenRates	Default generic priors
DEFAULT_PRIOR_STRENGTH	1_000_000	Default ridge strength λ
BUILTIN_TOKEN_RATES	{model: TokenRates}	Shipped baseline for known models

---

License

MIT — see LICENSE.