Tokenomics

Benchmarking suite for OpenAI-compatible inference servers. Measures throughput, latency, and steady-state performance.

Install

pip install tokenomics

From source

git clone https://github.com/tugot17/tokenomics.git
cd tokenomics
uv venv --python 3.12 --seed && source .venv/bin/activate
uv pip install -e .

Completion Benchmark

Sends chat completion requests to any OpenAI-compatible server and records per-request and system-wide metrics. Requests are non-streaming by default (max throughput); --stream adds TTFT and per-token metrics.

Every run sweeps concurrency and writes one result JSON per sweep point. Two execution modes, mutually exclusive:

• --max-concurrency 1,2,4,… — sustained: holds concurrency constant (realistic production numbers).
• --batch-sizes 1,2,4,… — burst: fires each batch at once (peak throughput).

tokenomics completion --model your-model \
  --scenario "D(1024,256)" \
  --max-concurrency 1,2,4,8,16,32,64,128 \
  --results-dir results/

Building requests

Three ways to produce request content, all swept over concurrency the same way:

Synthetic (--scenario)

A scenario sets input/output token counts; the prompt is built by concatenating random dataset snippets (with replacement) until it reaches the input budget.

The snippet source defaults to a bundled AIME dataset; override with --dataset-config (see Dataset config).

Dataset replay (--replay-dataset)

Sends each dataset row verbatim as one request and walks the dataset in order, instead of synthesizing prompts. The prompt set is pinned (same rows, same order across every concurrency level and run), so results line up example by example.

tokenomics completion --model your-model \
  --replay-dataset --dataset-config examples/dataset_configs/humaneval.json \
  --max-concurrency 1,2,4,8,16,32 --max-tokens 1024 \
  --results-dir results/humaneval/

• Sustained mode only; --scenario is ignored.
• --num-prompts N caps the walk to the first N rows (default: whole dataset).
• List prompt columns (MT-Bench prompt, Arena-Hard turns) are reduced to the first turn — MT-Bench runs single-turn.

Bundled configs under examples/dataset_configs/:

Vision replay: a config with an image_column (in addition to prompt_column) replays real image+text examples — each row's image(s) are attached to the request. This is the realistic VL workload (real prompts + real images, generating to natural EOS), e.g. for speculative-decoding evaluation. Bundled vision configs:

tokenomics completion --model your-vl-model \
  --replay-dataset --dataset-config examples/dataset_configs/chartqa.json \
  --max-concurrency 1,2,4,8,16 --results-dir results/chartqa/

Images (--num-images)

Attach images to any run — sent as OpenAI content parts (image_url base64 data: URIs, accepted by SGLang and vLLM) — turning it into a VL benchmark. Metrics and plotting are unchanged.

tokenomics completion --model your-vl-model \
  --num-images 5 --image-size 512x512 \
  --max-concurrency 1,2,4,8,16 \
  --results-dir results/vl_512x5/

Image runs default to a short workload (the images dominate, the text is padding): a fixed filler of --input-tokens (default 32) and --max-tokens 32 output.

• --image-size is N (square) or WxH (e.g. 1024x768, lowercase x); --input-tokens 0 = images only.
• Synthetic images are random-noise PNGs, seeded per request → unique (defeats the server's prefix/multimodal caches) yet reproducible. Noise is nearly incompressible (~MBs at 1024×1024), so keep size/count modest or the payload dominates.
• --input-tokens and --scenario are mutually exclusive; pass --scenario to put images on dataset-driven text instead.
• Sweep size/count/length by looping the command (one --results-dir each) and overlaying with plot-completion.

Dataset config

A JSON file with a source and (usually) a prompt_column. File paths are resolved relative to the config file.

{ "source": { "type": "huggingface", "path": "openai/gsm8k",
              "huggingface_kwargs": { "name": "main", "split": "test" } },
  "prompt_column": "question" }

source.type is huggingface, file (.txt/.csv/.json), or aime (bundled shortcut). Add an image_column (alongside prompt_column) to replay a vision dataset — its images (embedded PIL, {bytes/path}, or file paths) are encoded and attached per request. See examples/dataset_configs/ for more.

Output length & reproducibility

--ignore-eos makes every request generate exactly --max-tokens (EOS ignored), fixing output length so throughput isn't skewed by content-dependent token counts. Add it when comparing harnesses, servers, or configs; omit it for realistic, content-driven lengths. Supported by SGLang and vLLM — a no-op on servers that ignore the field.

--max-tokens defaults to 4096 (32 for image runs) and --temperature to 0.7. For fully reproducible runs, use --temperature 0 with --ignore-eos.

Key Options

Metrics

Per-request:

• TTFT — time to first token (streaming only)
• Decode throughput — output tokens/s per request (streaming only)
• TPOT — time per output token (streaming only)
• Per-request latency — end-to-end time per request

System-wide:

• End-to-end output throughput — total_output_tokens / wall_time
• Steady-state output throughput — median tok/s across time buckets where the batch is >= 80% full (streaming only)

Plotting

# Compare multiple benchmarks (overlays each results dir as its own line)
tokenomics plot-completion output.png results_dir1/ results_dir2/

Non-streaming (default) produces a 2-panel plot:

Streaming (--stream) produces a 6-panel dashboard:

Scoring Benchmark

tokenomics score benchmarks SGLang /v1/score.

Pointwise — one Yes/No score per candidate:

{
  "query": "<generated shared text>\n",
  "items": [
    "Action: Issue a refund.\nAnswer Yes or No:",
    "Action: Troubleshoot a software crash.\nAnswer Yes or No:",
    "Action: Cancel the subscription.\nAnswer Yes or No:"
  ]
}

Setwise — one A/B/C choice:

{
  "query": "<generated shared text>\n",
  "items": [
    "Choose an action:\nA) Issue a refund.\nB) Troubleshoot a software crash.\nC) Cancel the subscription.\nAnswer A, B, or C:"
  ]
}

Independent questions — one answer per question:

{
  "state": "<shared text generated by --state-tokens>",
  "questions": [
    {
      "prompt": "Is the customer requesting a refund? Answer yes or no.",
      "labels": [{"text": "yes"}, {"text": "no"}]
    },
    {
      "prompt": "Which team should handle this? A: Billing. B: Technical support. Answer A or B.",
      "labels": [{"text": "A"}, {"text": "B"}]
    }
  ]
}

The state shown above is generated by --state-tokens; the input file supplies candidates or questions. Tokenomics applies the model's chat template, splits the tokenized prompts into query/items, and resolves label IDs.

tokenomics score --model your-model --api-base http://localhost:30000/v1 \
  --formulation setwise --config examples/score_setwise.json \
  --dataset examples/data/score_candidate_sets.jsonl \
  --state-tokens 1024 --batch-sizes 1,8,32,128 \
  --num-runs 1 --warmup-runs 1 --results-dir results/scoring

The JSONL file defines the candidates or questions. --state-tokens generates shared text at each requested length; results measure throughput and latency.

tokenomics plot-score results/scoring scoring.png

Embedding Benchmark

Tests concurrent embedding throughput.

tokenomics embedding \
  --model Qwen/Qwen3-Embedding-4B \
  --sequence_lengths "200" \
  --batch_sizes "1,8,16,32,64,128,256,512" \
  --num_runs 3 \
  --results-dir embedding_results/

tokenomics plot-embedding embedding_results/ embedding_plot.png