Skip to main content

llm-token-visualizer for Python

Find the words an LLM answer was unsure about. Ask the API for token logprobs, pass the response to score, and get back the low-confidence spans, an HTML report and a summary. No extra API calls, no second model, no Python dependencies: the parsing and detection are the llm-token-visualizer Rust crate, compiled into the wheel.

pip install llm-token-visualizer

Score an OpenAI answer

from openai import OpenAI
import llm_token_visualizer as ltv
resp = OpenAI().chat.completions.create(model="gpt-4.1-mini", logprobs=True, top_logprobs=5,
    messages=[{"role": "user", "content": "Who won the 1998 Tour de France?"}])
result = ltv.score(resp)
print(result.summary())

logprobs=True is what makes this work. top_logprobs=5 is optional and adds the alternatives the model was weighing at each weak word. For an answer of "The 1998 Tour de France was won by Marco Pantani." where the model hesitated on the name (the test fixture python/tests/fixtures/chat_tour.json), the summary is:

1 low-confidence span below p=0.50 in 14 tokens (mean p=0.96, perplexity 1.07):
  [34:40] " Marco" p=0.41 at " Marco"; model also considered " Jan" (0.38), " Bjarne" (0.12), " Marc" (0.01)

result.to_html() gives the same answer as a page with every token shaded by confidence: open("report.html", "w", encoding="utf-8").write(result.to_html()).

A low probability is a signal, not proof. A model can be confidently wrong, and it can be unsure about wording while the facts are right. Use the spans to decide what to check, retry or show with a warning.

What score returns

score(resp, threshold=0.5, *, choice=0) returns a Score:

spans() The low-confidence spans, in order.
to_html(title=None) A self-contained HTML page (no scripts, no external files) with every token shaded by confidence. Hover a token for its probability and the alternatives.
to_markdown(title=None) The same report as Markdown, for a pull request or issue comment.
summary() One header line, then one line per span.
to_json() The report as JSON, the shape the CLI's --format json writes.
text The answer, tokens joined. Span offsets index into it.
flagged True when there is at least one span.
mean_prob, perplexity, token_count, threshold, model Whole-answer numbers.

Each Span has:

start, end Character offsets into Score.text: result.text[s.start:s.end] == s.text.
text The span text.
min_logprob, mean_logprob Lowest and mean natural-log probability of the span's tokens.
min_prob, weakest_token The word token that got the span flagged, and its probability.
alternatives (token, probability) pairs the model weighed at that token, most likely first. Empty without top_logprobs.
entropy_bits Entropy of those alternatives in bits (a lower bound), or None.
token_start, token_end The same span as token indices.
describe(), to_dict() One line of text, or a plain dict.

Two spans are equal (==) when every field is equal, so score(r).spans() == score(r).spans(). Spans are not hashable; use s.to_dict() or (s.start, s.end) as a key.

A word is flagged when any of its tokens with letters or digits has a probability strictly below threshold. Neighbouring flagged words merge into one span, and punctuation or whitespace alone never triggers a flag.

text is the tokens joined. It matches message.content except where the API splits one character (an emoji, some CJK) across tokens.

Gate a CI or eval run

gate(resp, threshold) is True when nothing in the answer is below the threshold:

import llm_token_visualizer as ltv

def test_capital_answer_is_confident(recorded_response):
    assert ltv.gate(recorded_response, 0.4), ltv.score(recorded_response, 0.4).summary()

What it reads

  • OpenAI SDK objects: ChatCompletion, legacy Completion, a single Choice, its logprobs, or the token list resp.choices[0].logprobs.content. Lists and dicts that hold SDK objects work too.
  • The same responses as a dict, a JSON string or bytes. That covers any OpenAI-compatible server that returns logprobs: vLLM, OpenRouter, Together, Ollama.
  • The Completions shape (tokens / token_logprobs / top_logprobs), including the null first logprob of an echo=True request (kept in the text, never flagged).
  • vLLM offline generation: a RequestOutput or CompletionOutput from LLM.generate(..., SamplingParams(logprobs=5)).
  • Google Gemini generateContent responses made with responseLogprobs: true.

With n > 1, pick the choice with score(resp, choice=1).

Errors

  • No logprobs in the response, or logprobs is None: NoLogprobsError, saying to request the completion with logprobs=True and naming the choice (choices[1].logprobs is null for choice=1). Catch it to retry with logprobs on.
  • An API error body, invalid JSON, bytes that are not UTF-8, or a malformed logprobs block: LogprobsError.
  • A threshold outside 0 to 1, or a choice the response does not have: ValueError.
  • An input it cannot read (not a response, dict, string or vLLM output): TypeError.

NoLogprobsError is a subclass of LogprobsError, and both are ValueErrors.

try:
    result = ltv.score(resp)
except ltv.NoLogprobsError:
    resp = client.chat.completions.create(**request, logprobs=True, top_logprobs=5)
    result = ltv.score(resp)

Platforms

One abi3 wheel per platform covers CPython 3.9 and newer: Linux x86_64 and aarch64 (manylinux and musllinux), macOS x86_64 and arm64, Windows x86_64. Elsewhere pip builds from the source distribution, which needs a Rust toolchain.

Build from source

python -m venv .venv && . .venv/bin/activate
pip install maturin pytest
maturin develop --release
pytest

The bindings are in python/ (python/src/lib.rs and python/llm_token_visualizer/); pyproject.toml is at the repository root. Wheels are built and published to PyPI by .github/workflows/python.yml on tags named py-v<version>.

If that workflow cannot run, a release can be uploaded by hand with a PyPI API token. This builds only the wheel for the machine it runs on plus the sdist, so other platforms install from the sdist (which needs Rust) until their wheels are uploaded:

pip install maturin
maturin build --release --out dist
maturin sdist --out dist
MATURIN_PYPI_TOKEN=pypi-... maturin upload dist/*

A version uploaded to PyPI cannot be replaced, only yanked, so run pytest against the built wheel first.

Metadata

Release files for llm-token-visualizer 0.7.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for llm-token-visualizer 0.7.0
File Size Uploaded Provenance Actions
llm_token_visualizer-0.7.0.tar.gz 132.1 kB With provenance Details for llm_token_visualizer-0.7.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for llm-token-visualizer 0.7.0
File Actions
llm_token_visualizer-0.7.0-cp39-abi3-win_amd64.whl CPython 3.9 abi3 Windows x86-64 Details for llm_token_visualizer-0.7.0-cp39-abi3-win_amd64.whl
llm_token_visualizer-0.7.0-cp39-abi3-musllinux_1_2_x86_64.whl CPython 3.9 abi3 Linux musl 1.2+ x86-64 Details for llm_token_visualizer-0.7.0-cp39-abi3-musllinux_1_2_x86_64.whl
llm_token_visualizer-0.7.0-cp39-abi3-musllinux_1_2_aarch64.whl CPython 3.9 abi3 Linux musl 1.2+ ARM64 Details for llm_token_visualizer-0.7.0-cp39-abi3-musllinux_1_2_aarch64.whl
llm_token_visualizer-0.7.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.9 abi3 Linux glibc 2.17+ x86-64 Details for llm_token_visualizer-0.7.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
llm_token_visualizer-0.7.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl CPython 3.9 abi3 Linux glibc 2.17+ ARM64 Details for llm_token_visualizer-0.7.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
llm_token_visualizer-0.7.0-cp39-abi3-macosx_11_0_arm64.whl CPython 3.9 abi3 macOS 11.0+ ARM64 Details for llm_token_visualizer-0.7.0-cp39-abi3-macosx_11_0_arm64.whl
llm_token_visualizer-0.7.0-cp39-abi3-macosx_10_12_x86_64.whl CPython 3.9 abi3 macOS 10.12+ x86-64 Details for llm_token_visualizer-0.7.0-cp39-abi3-macosx_10_12_x86_64.whl

Total release size: 3.4 MB

Release files / llm_token_visualizer-0.7.0.tar.gz

Download URL llm_token_visualizer-0.7.0.tar.gz
Size 132.1 kB
Tags Source
SHA-256 checksum
How to use checksums
176a16f021d4e336e33dc64f7410ee91ab971d18e0f3c8f7845c8ddcda162305
BLAKE2b-256 checksum
How to use checksums
a989f7b995d3f700d6722c7202306afacf2b40a68e422797273399aad805f948
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

Release files / llm_token_visualizer-0.7.0-cp39-abi3-win_amd64.whl

Download URL llm_token_visualizer-0.7.0-cp39-abi3-win_amd64.whl
Size 293.7 kB
Tags CPython 3.9 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
9deb45b6eeedd9c5783b699819de1a3fc6b10e27d4b4898db73572e7bf0e724c
BLAKE2b-256 checksum
How to use checksums
cdc7656b7044a7ac0803374471763cde9e59563855d368fbdf0c4b4da3c21d8c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

Release files / llm_token_visualizer-0.7.0-cp39-abi3-musllinux_1_2_x86_64.whl

Download URL llm_token_visualizer-0.7.0-cp39-abi3-musllinux_1_2_x86_64.whl
Size 654.6 kB
Tags CPython 3.9 Linux musl 1.2+ x86-64 abi3
SHA-256 checksum
How to use checksums
f4ffcf777922896bc2838d200f63d1243b5aa03ef44e4ab8467b6e0b3351a262
BLAKE2b-256 checksum
How to use checksums
d975cf7b2daf81ed0af06dde35701294d89ded624ddf6163c939f7d3a8c8d4e8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

Release files / llm_token_visualizer-0.7.0-cp39-abi3-musllinux_1_2_aarch64.whl

Download URL llm_token_visualizer-0.7.0-cp39-abi3-musllinux_1_2_aarch64.whl
Size 613.7 kB
Tags CPython 3.9 Linux musl 1.2+ ARM64 abi3
SHA-256 checksum
How to use checksums
4e72c1daf6ea5a078d2123be09cd9cf1d2cb5a5f115f56b21e7861937726a3d2
BLAKE2b-256 checksum
How to use checksums
d6d04af992c2248320edcd479d353dd9584071c0418b0969ac9c94bd6e055b2f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

Release files / llm_token_visualizer-0.7.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL llm_token_visualizer-0.7.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 451.8 kB
Tags CPython 3.9 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
2fe221f5991566d3d02418990243a13599b02a7d64607c9f51e6bde76019be3b
BLAKE2b-256 checksum
How to use checksums
2c37adad595c56a009aba7d5fee0a849111d2875a4e471558d51eb3f4e349245
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

Release files / llm_token_visualizer-0.7.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl

Download URL llm_token_visualizer-0.7.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Size 436.5 kB
Tags CPython 3.9 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum
How to use checksums
6da809a7df6132e5c9325bb9c0d14c1e6206a39925acfe55443a5b316d021879
BLAKE2b-256 checksum
How to use checksums
c0ff085d71f16de1b72191e98b0fc2000b1bb1a08763ff1aa0889a27814167d9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

Release files / llm_token_visualizer-0.7.0-cp39-abi3-macosx_11_0_arm64.whl

Download URL llm_token_visualizer-0.7.0-cp39-abi3-macosx_11_0_arm64.whl
Size 396.4 kB
Tags CPython 3.9 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
cf2457f3af67c68c56125fe50971b7f3718b4daa47d8dad90e6d31bc95c611dc
BLAKE2b-256 checksum
How to use checksums
e2d1f179da320c27897d5671104f98c3d67323d755a76bfaf48b68d731416ffc
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

Release files / llm_token_visualizer-0.7.0-cp39-abi3-macosx_10_12_x86_64.whl

Download URL llm_token_visualizer-0.7.0-cp39-abi3-macosx_10_12_x86_64.whl
Size 405.9 kB
Tags CPython 3.9 abi3 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
7f9507a5fe0c0a9b5af253be1209c49421a9551197084de88d5bcfbfccb79b66
BLAKE2b-256 checksum
How to use checksums
929586f300b8b40cf54d97ed552312950e0735b09bbf302615f63db300a5d928
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

Security for llm-token-visualizer 0.7.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 8 files in this release have been uploaded with provenance metadata.

Select from 8 files

Attestation for llm_token_visualizer-0.7.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.7.0 This release

8 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page