toklen

Count tokens with a model's real Hugging Face tokenizer. Prints an integer.

npx toklen openai-community/gpt2 "hello world"   # 2
uvx toklen openai-community/gpt2 "hello world"   # 2

One repo, two packages, same number. Node uses @huggingface/tokenizers (pure JS, no dependencies), Python uses the Rust tokenizers bindings. One dependency each. No transformers, no PyTorch, no ONNX.

CLI

usage: toklen [-s] <model> [text]

Print the number of tokens in TEXT under a Hugging Face model's tokenizer.
Reads stdin when TEXT is omitted or "-".

options:
  -s, --special  also count the special tokens the model adds (e.g. BOS/EOS)
  -h, --help     show this message
  -v, --version  show the version

environment:
  HF_TOKEN       token for gated or private models
  HF_ENDPOINT    Hugging Face endpoint (default https://huggingface.co)
  TOKLEN_CACHE   tokenizer cache directory (default ~/.cache/toklen)

<model> is a Hub slug, a slug pinned to a revision (Qwen/Qwen3-8B@refs/pr/1), or a path to a local tokenizer.json.

toklen Qwen/Qwen3-8B < README.md
git diff | toklen openai-community/gpt2
toklen -s google-bert/bert-base-uncased "hello world"   # 4, with [CLS] and [SEP]

The first run for a model downloads its tokenizer.json to the cache. Later runs need no network.

Library

import { count, loadTokenizer } from "toklen";

await count("openai-community/gpt2", "hello world");                  // 2
await count("openai-community/gpt2", "hello world", { special: true });
const tokenizer = await loadTokenizer("openai-community/gpt2");

from toklen import count, load_tokenizer

count("openai-community/gpt2", "hello world")                  # 2
count("openai-community/gpt2", "hello world", special=True)
tokenizer = load_tokenizer("openai-community/gpt2")

loadTokenizer returns the underlying tokenizer object, so you can get the tokens themselves. Both languages raise ToklenError for an unknown model, a gated repo, or a failed download.

What gets counted

Exactly the bytes you pass. echo hi | toklen gpt2 counts the newline echo adds.

No special tokens unless you pass -s. Chat models add theirs through a chat template rather than the raw text, so the plain count is the one to budget against.

No padding or truncation, even when the tokenizer.json asks for them. sentence-transformers/all-MiniLM-L6-v2 pads to 128, which would report 128 for every input.

Parity

tests/parity.sh runs both CLIs over 6 models, 10 awkward texts (empty, whitespace, CJK, ZWJ emoji, fullwidth, combining accents), both flag modes, argv against stdin, and the usage and error exit codes. 134 checks, and it fails on any disagreement.

npm install && uv venv && uv pip install -e .
TOKLEN_PY=".venv/bin/python -m toklen" tests/parity.sh

License

MIT