llm-token-visualizer for Python
Find the words an LLM answer was unsure about. Ask the API for token logprobs, pass the response to score, and get back the low-confidence spans, an HTML report and a summary. No extra API calls, no second model, no Python dependencies: the parsing and detection are the llm-token-visualizer Rust crate, compiled into the wheel.
pip install llm-token-visualizer
Score an OpenAI answer
from openai import OpenAI
import llm_token_visualizer as ltv
resp = OpenAI().chat.completions.create(model="gpt-4.1-mini", logprobs=True, top_logprobs=5,
messages=[{"role": "user", "content": "Who won the 1998 Tour de France?"}])
result = ltv.score(resp)
print(result.summary())
logprobs=True is what makes this work. top_logprobs=5 is optional and adds the alternatives the model was weighing at each weak word. For an answer of "The 1998 Tour de France was won by Marco Pantani." where the model hesitated on the name (the test fixture python/tests/fixtures/chat_tour.json), the summary is:
1 low-confidence span below p=0.50 in 14 tokens (mean p=0.96, perplexity 1.07):
[34:40] " Marco" p=0.41 at " Marco"; model also considered " Jan" (0.38), " Bjarne" (0.12), " Marc" (0.01)
result.to_html() gives the same answer as a page with every token shaded by confidence: open("report.html", "w", encoding="utf-8").write(result.to_html()).
A low probability is a signal, not proof. A model can be confidently wrong, and it can be unsure about wording while the facts are right. Use the spans to decide what to check, retry or show with a warning.
What score returns
score(resp, threshold=0.5, *, choice=0) returns a Score:
spans() |
The low-confidence spans, in order. |
to_html(title=None) |
A self-contained HTML page (no scripts, no external files) with every token shaded by confidence. Hover a token for its probability and the alternatives. |
to_markdown(title=None) |
The same report as Markdown, for a pull request or issue comment. |
summary() |
One header line, then one line per span. |
to_json() |
The report as JSON, the shape the CLI's --format json writes. |
text |
The answer, tokens joined. Span offsets index into it. |
flagged |
True when there is at least one span. |
mean_prob, perplexity, token_count, threshold, model |
Whole-answer numbers. |
Each Span has:
start, end |
Character offsets into Score.text: result.text[s.start:s.end] == s.text. |
text |
The span text. |
min_logprob, mean_logprob |
Lowest and mean natural-log probability of the span's tokens. |
min_prob, weakest_token |
The word token that got the span flagged, and its probability. |
alternatives |
(token, probability) pairs the model weighed at that token, most likely first. Empty without top_logprobs. |
entropy_bits |
Entropy of those alternatives in bits (a lower bound), or None. |
token_start, token_end |
The same span as token indices. |
describe(), to_dict() |
One line of text, or a plain dict. |
Two spans are equal (==) when every field is equal, so score(r).spans() == score(r).spans(). Spans are not hashable; use s.to_dict() or (s.start, s.end) as a key.
A word is flagged when any of its tokens with letters or digits has a probability strictly below threshold. Neighbouring flagged words merge into one span, and punctuation or whitespace alone never triggers a flag.
text is the tokens joined. It matches message.content except where the API splits one character (an emoji, some CJK) across tokens.
Gate a CI or eval run
gate(resp, threshold) is True when nothing in the answer is below the threshold:
import llm_token_visualizer as ltv
def test_capital_answer_is_confident(recorded_response):
assert ltv.gate(recorded_response, 0.4), ltv.score(recorded_response, 0.4).summary()
What it reads
- OpenAI SDK objects:
ChatCompletion, legacyCompletion, a singleChoice, itslogprobs, or the token listresp.choices[0].logprobs.content. Lists and dicts that hold SDK objects work too. - The same responses as a dict, a JSON string or bytes. That covers any OpenAI-compatible server that returns logprobs: vLLM, OpenRouter, Together, Ollama.
- The Completions shape (
tokens/token_logprobs/top_logprobs), including thenullfirst logprob of anecho=Truerequest (kept in the text, never flagged). - vLLM offline generation: a
RequestOutputorCompletionOutputfromLLM.generate(..., SamplingParams(logprobs=5)). - Google Gemini
generateContentresponses made withresponseLogprobs: true.
With n > 1, pick the choice with score(resp, choice=1).
Errors
- No logprobs in the response, or
logprobsisNone:NoLogprobsError, saying to request the completion withlogprobs=Trueand naming the choice (choices[1].logprobs is nullforchoice=1). Catch it to retry with logprobs on. - An API error body, invalid JSON, bytes that are not UTF-8, or a malformed logprobs block:
LogprobsError. - A threshold outside 0 to 1, or a
choicethe response does not have:ValueError. - An input it cannot read (not a response, dict, string or vLLM output):
TypeError.
NoLogprobsError is a subclass of LogprobsError, and both are ValueErrors.
try:
result = ltv.score(resp)
except ltv.NoLogprobsError:
resp = client.chat.completions.create(**request, logprobs=True, top_logprobs=5)
result = ltv.score(resp)
Platforms
One abi3 wheel per platform covers CPython 3.9 and newer: Linux x86_64 and aarch64 (manylinux and musllinux), macOS x86_64 and arm64, Windows x86_64. Elsewhere pip builds from the source distribution, which needs a Rust toolchain.
Build from source
python -m venv .venv && . .venv/bin/activate
pip install maturin pytest
maturin develop --release
pytest
The bindings are in python/ (python/src/lib.rs and python/llm_token_visualizer/); pyproject.toml is at the repository root. Wheels are built and published to PyPI by .github/workflows/python.yml on tags named py-v<version>.
If that workflow cannot run, a release can be uploaded by hand with a PyPI API token. This builds only the wheel for the machine it runs on plus the sdist, so other platforms install from the sdist (which needs Rust) until their wheels are uploaded:
pip install maturin
maturin build --release --out dist
maturin sdist --out dist
MATURIN_PYPI_TOKEN=pypi-... maturin upload dist/*
A version uploaded to PyPI cannot be replaced, only yanked, so run pytest against the built wheel first.
Metadata
Release files for llm-token-visualizer 0.7.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| llm_token_visualizer-0.7.0.tar.gz | 132.1 kB | With provenance | Details for llm_token_visualizer-0.7.0.tar.gz |
Built distributions (wheels)
Total release size: 3.4 MB
Release files / llm_token_visualizer-0.7.0.tar.gz
| Download URL | llm_token_visualizer-0.7.0.tar.gz |
|---|---|
| Size | 132.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
176a16f021d4e336e33dc64f7410ee91ab971d18e0f3c8f7845c8ddcda162305
|
|
BLAKE2b-256 checksum How to use checksums |
a989f7b995d3f700d6722c7202306afacf2b40a68e422797273399aad805f948
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logRelease files / llm_token_visualizer-0.7.0-cp39-abi3-win_amd64.whl
| Download URL | llm_token_visualizer-0.7.0-cp39-abi3-win_amd64.whl |
|---|---|
| Size | 293.7 kB |
| Tags | CPython 3.9 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
9deb45b6eeedd9c5783b699819de1a3fc6b10e27d4b4898db73572e7bf0e724c
|
|
BLAKE2b-256 checksum How to use checksums |
cdc7656b7044a7ac0803374471763cde9e59563855d368fbdf0c4b4da3c21d8c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logRelease files / llm_token_visualizer-0.7.0-cp39-abi3-musllinux_1_2_x86_64.whl
| Download URL | llm_token_visualizer-0.7.0-cp39-abi3-musllinux_1_2_x86_64.whl |
|---|---|
| Size | 654.6 kB |
| Tags | CPython 3.9 Linux musl 1.2+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
f4ffcf777922896bc2838d200f63d1243b5aa03ef44e4ab8467b6e0b3351a262
|
|
BLAKE2b-256 checksum How to use checksums |
d975cf7b2daf81ed0af06dde35701294d89ded624ddf6163c939f7d3a8c8d4e8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logRelease files / llm_token_visualizer-0.7.0-cp39-abi3-musllinux_1_2_aarch64.whl
| Download URL | llm_token_visualizer-0.7.0-cp39-abi3-musllinux_1_2_aarch64.whl |
|---|---|
| Size | 613.7 kB |
| Tags | CPython 3.9 Linux musl 1.2+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
4e72c1daf6ea5a078d2123be09cd9cf1d2cb5a5f115f56b21e7861937726a3d2
|
|
BLAKE2b-256 checksum How to use checksums |
d6d04af992c2248320edcd479d353dd9584071c0418b0969ac9c94bd6e055b2f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logRelease files / llm_token_visualizer-0.7.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
| Download URL | llm_token_visualizer-0.7.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl |
|---|---|
| Size | 451.8 kB |
| Tags | CPython 3.9 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
2fe221f5991566d3d02418990243a13599b02a7d64607c9f51e6bde76019be3b
|
|
BLAKE2b-256 checksum How to use checksums |
2c37adad595c56a009aba7d5fee0a849111d2875a4e471558d51eb3f4e349245
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logRelease files / llm_token_visualizer-0.7.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
| Download URL | llm_token_visualizer-0.7.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl |
|---|---|
| Size | 436.5 kB |
| Tags | CPython 3.9 Linux glibc 2.17+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
6da809a7df6132e5c9325bb9c0d14c1e6206a39925acfe55443a5b316d021879
|
|
BLAKE2b-256 checksum How to use checksums |
c0ff085d71f16de1b72191e98b0fc2000b1bb1a08763ff1aa0889a27814167d9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logRelease files / llm_token_visualizer-0.7.0-cp39-abi3-macosx_11_0_arm64.whl
| Download URL | llm_token_visualizer-0.7.0-cp39-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 396.4 kB |
| Tags | CPython 3.9 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
cf2457f3af67c68c56125fe50971b7f3718b4daa47d8dad90e6d31bc95c611dc
|
|
BLAKE2b-256 checksum How to use checksums |
e2d1f179da320c27897d5671104f98c3d67323d755a76bfaf48b68d731416ffc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logRelease files / llm_token_visualizer-0.7.0-cp39-abi3-macosx_10_12_x86_64.whl
| Download URL | llm_token_visualizer-0.7.0-cp39-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 405.9 kB |
| Tags | CPython 3.9 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
7f9507a5fe0c0a9b5af253be1209c49421a9551197084de88d5bcfbfccb79b66
|
|
BLAKE2b-256 checksum How to use checksums |
929586f300b8b40cf54d97ed552312950e0735b09bbf302615f63db300a5d928
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logSecurity for llm-token-visualizer 0.7.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 8 files in this release have been uploaded with provenance metadata.
Attestation for llm_token_visualizer-0.7.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.