Most evaluation tools end with a score and a table of failures. EvalVitals starts there. It can explore raw eval logs, find recurring patterns, propose falsifiable explanations, test them on held-out cases, and validate candidate interventions against the unchanged baseline.

Use the whole investigation loop—or take only the layer you need:

• Explore existing results: point a coding agent at arbitrary JSON/JSONL logs and get observations, charts, tables, and testable hypotheses.
• Investigate a target model: adaptively probe failures, diagnose systematic modes, verify hypotheses, and test interventions.
• Run one focused analyzer: use the same sklearn-like interface for black-box APIs and white-box local models.

flowchart LR
    A[Eval logs or target model] --> B[Probe]
    B --> C[Explore patterns]
    C --> D[Diagnose]
    D --> E{Held-out verification}
    E -->|supported| F[Test interventions]
    E -->|refuted| B
    F --> G[Validated fix or honest inconclusive result]

Quickstart: Analyze Your Eval Logs

Install EvalVitals with the report dashboard:

pip install "evalvitals[dashboard]"

Then point it at a file or directory of JSON/JSONL results:

evalvitals explore ./results \
  --backend codex \
  -q "What distinguishes failed cases from successful ones?" \
  --dashboard

codex can be replaced with claude_code, opencode, gemini_cli, kimi_cli, or antigravity. The selected coding-agent CLI must be installed and authenticated separately.

Prefer a browser? evalvitals web serves a local data-analysis workbench: drop a .zip containing JSON/JSONL/CSV/TSV/Parquet/Excel data and/or images, PDFs, audio, or video. Each upload becomes a persistent data thread: the page shows ingestion and M2/M3 progress, renders M2 as soon as it finishes, and accepts artifact-grounded follow-up questions without uploading again.

EvalVitals writes an auditable analysis bundle instead of returning only prose:

evalvitals_explore_output/
├── exploratory_report.json   # observations, candidate signals, hypotheses
├── records.json              # normalized records used by the analysis
├── figures/                  # rendered charts
├── tables/                   # analysis-ready tables
└── analysis.py               # the generated code that was actually run

A real bundled run: on the synthetic-yield example, Explore identified temperature as the strongest observed correlate (r = 0.86), separated that finding from weak pressure evidence (r = -0.21), and proposed mechanism-level hypotheses for a later confirmatory experiment. See the reproducible example →

Already have your own analysis code? Use the analyzer toolkit directly, or feed the resulting cases into the full diagnosis loop. EvalVitals does not require you to replace your existing eval or observability stack.

What Makes It Different

Statistical gates use paired tests and e-values, including multiplicity control when several hypotheses or fixes are tried. A run may end inconclusive; EvalVitals does not turn weak evidence into a success verdict.

Three Ways to Use EvalVitals

1. Explore — raw results to testable hypotheses

evalvitals explore recursively samples arbitrary JSON/JSONL shapes. The coding agent performs exploratory data analysis; the host records generated code, adjudicates host-checkable statistics, renders figures, and proposes 1–3 falsifiable hypotheses.

Explore guide →

2. Investigate — failures to verified interventions

VLDiagnoseLoop chains the full workflow:

M1 targeted probes
 → M2 exploratory and statistical analysis
 → M3 diagnosis hypotheses
 → M5 held-out hypothesis verification
 → M4 surgery and tiered fixes

Interventions can range from prompt changes and scaffolds to read/write access to model internals. Each candidate is evaluated against the unmodified baseline; automatic escalation happens only when explicitly enabled.

Full-loop quickstart → · Intervention guide →

3. Analyze — one model, one question

Every registered analyzer follows the same call shape:

from evalvitals import Capability, compose
from evalvitals.analyzers.attention.summary import AttentionAnalyzer

model = compose(
    "qwen2.5-7b-instruct",
    "hf_local",
    want={Capability.ATTENTION},
)

result = AttentionAnalyzer(layer=-1, top_k=5).run(
    model, "The Eiffel Tower is in"
)

print(result.summary())

The analyzer zoo includes attention, uncertainty, hallucination, attribution, logit-lens, representation-geometry, and agent-trajectory analysis.

Browse the Analyzer Zoo →

Installation

The core install stays lightweight—no Torch required:

pip install evalvitals

Add only the capabilities you need:

pip install "evalvitals[api]"        # OpenAI-compatible API models
pip install "evalvitals[local]"      # local Hugging Face models + Torch
pip install "evalvitals[interp]"     # interpretability toolchains
pip install "evalvitals[viz]"        # plots
pip install "evalvitals[dashboard]"  # Streamlit reports
pip install "evalvitals[stats]"      # inferential statistics

For development:

git clone https://github.com/evalvitals/evalvitals.git
cd evalvitals
pip install -e ".[dev]"
pytest -m "not gpu"

Architecture in One Minute

Model identity is separate from runtime, and analyzers declare the capabilities they need. The same model spec can run through a black-box API or a white-box local backend; only the available capability set changes.

Read the architecture guide →

Reproducible Examples

See all examples →

Documentation

• Quickstart
• Exploratory Analysis
• Intervention & Verification
• Analyzer Zoo
• Architecture
• Extending EvalVitals
• Roadmap

Project Status

EvalVitals is an early-stage research toolkit. Interfaces may evolve, and some full-loop examples require model weights, a GPU, or an external coding-agent CLI. Bug reports, reproducible failure cases, analyzer contributions, and evaluation integrations are welcome.

If EvalVitals helps you understand a model failure, consider starring the repo and sharing the smallest reproducible case—it makes the toolkit better for the next investigation.