LLM-Analyzer

Universal, provider-agnostic local LLM response analyzer.

LLM-Analyzer captures LLM responses, extracts available metadata, stores telemetry locally in SQLite, and provides a local dashboard.

Features

• Provider-agnostic response capture
• Model and provider metadata extraction when available
• Token usage extraction when available
• Latency measurement
• Error capture
• Temporary local SQLite storage
• Automatic local dashboard at http://127.0.0.1:1918
• Optional response evaluation
• CSV export
• Python API and CLI

The analyzer does not require an OpenAI, Gemini, Groq, Claude, or other provider SDK.

Install

pip install llm-analyzer

Basic usage

from llm_analyzer import analyze

response = my_llm_call(prompt)

analyze(response)

return response

Or measure the complete call automatically:

from llm_analyzer import analyze_call

response = analyze_call(my_llm_call, prompt)

return response

The first analysis call automatically starts the local dashboard.

Open:

http://127.0.0.1:1918

Local storage

By default telemetry is stored in:

.llm-analyzer/telemetry.db

Captured data is temporary by default and uses a 7-day retention period.

You can change the retention period:

from llm_analyzer import Analyzer

analyzer = Analyzer(retention_days=3)

Or with an environment variable:

LLM_ANALYZER_RETENTION_DAYS=3

Disable automatic dashboard startup

Set:

LLM_ANALYZER_AUTO_DASHBOARD=0

CLI

Start the dashboard manually:

llm-analyzer dashboard

Print the database path:

llm-analyzer path

Evaluation

Evaluation is optional. The package can use supplied prompt/reference/context evidence or a custom evaluator.

Quality metrics such as accuracy, hallucination, bias, and fairness should be treated as evaluation results, not as metadata automatically inferred from an arbitrary LLM response.

License

MIT