Verdict Inspect

PyPI distribution: cognifity-verdict-inspect. Command: verdict-inspect.

One-shot drift analysis on a chat export. Drop in a conversations.json from ChatGPT, a Claude.ai export, a supported agent-session JSONL file, or an OpenAI-format messages dump, and get back a local drift / quality report.

Why this exists

Continuous LLM observability (the SDK + monitoring path) is the right answer for production agent traffic. But before a team commits to instrumenting their stack, they want to know: does Verdict actually find anything interesting in my data?

verdict-inspect runs against a file the user already has on their laptop. Structural and embedding analysis stay local; the optional judge has a separate privacy boundary described below.

Usage

In a full Verdict installation, the same one-off workflow is available under Evaluate → Inspect JSON. Paste JSON or choose a file, run the analysis, and download the result as JSON. The dashboard accepts at most 4 MiB per request, runs analysis on the Verdict host, does not write the upload or report to the Verdict store, and keeps semantic analysis and the external judge off until selected. Enabling the judge also requires an explicit data-egress confirmation.

The command-line interface remains available for larger local exports and Markdown reports:

# Auto-detect format
verdict-inspect analyze ~/Downloads/conversations.json

# Force a format
verdict-inspect analyze --format chatgpt ~/Downloads/conversations.json

# Specify report output
verdict-inspect analyze --report ./drift_report.md ~/Downloads/chatlog.jsonl

# JSON output for piping
verdict-inspect analyze --json ~/Downloads/conversations.json | jq .

Supported formats (v0)

• ChatGPT data export — the conversations.json from Settings → Data Controls → Export
• Claude.ai data export — the conversations.json from Settings → Account → Export
• Generic OpenAI messages JSONL — one JSON object per line, each with messages: [{role, content}]
• Agent-session JSONL — type-tagged local agent session logs
• Auto-detect — looks at file structure and picks a parser

Planned (v1): Cursor .cursor/chats/, Gemini Takeout, LangChain message history files, Llama Index conversation logs.

What you get back

For a file with enough substantive assistant turns:

1. Semantic drift — embedding-distribution shifts across temporal windows
2. Judge sample — PASS/FAIL by dimension on stride-sampled turns (requires ANTHROPIC_API_KEY)
3. Structural metrics — response length, hedge density, refusal rate, apology rate per window

Semantic drift runs key-free. By default it tries sentence-transformers/all-MiniLM-L6-v2, then falls back to the built-in HashingEmbedder if the dependency/model is unavailable. That fallback detects lexical embedding-distribution changes; it is not a semantic model, and the report labels it explicitly. Install the local semantic embedder with pip install "cognifity-verdict-eval[semantic]".

Triggered and non-triggered semantic rows use the same L2-normalized detector statistics. Each comparison embeds its current and baseline windows once; the report does not re-embed or independently recompute non-triggered rows.

Turns with fewer than 10 assistant-response words are excluded from windowed analysis. At least 16 substantive turns are required for a two-window comparison; 24 create the default early/middle/late split. Each window's judge sample is capped at 25 turns. Treat small-window output as exploratory rather than calibrated production evidence. Chat exports do not contain retrieved context, so the default context-required groundedness dimension is shown as n/a and is not sent to the judge.

Privacy

Structural metrics and embedding inference run on your machine. The first MiniLM run may download model weights, but it does not upload the analyzed conversation. If ANTHROPIC_API_KEY is set and the judge is enabled, verdict-inspect sends stride-sampled user and assistant text (up to 4,000 characters each) to the configured Anthropic model. Anthropic credentials and data-handling terms apply. Pass --no-judge or omit the key to keep conversation content local and receive structural plus embedding analysis only. The v0 inspect judge is Anthropic-only.