examen 0.3.0

Self-administered comprehension quiz for AI-assisted code

Examen

It's like an exam. For your own code. You will get a D.

I built this because I kept failing my own code reviews. I'd merge something Claude wrote, walk into standup the next day, get asked "why did you handle the edge case that way?" and freeze. Examen runs a 5-question quiz on your own repo. Each question shows you the code, the answer options, and a 1–5 confidence chip on one screen. You submit both together, so your gut goes on the record at the same time as your answer.

The results screen plots a heatmap: knowledge × calibration. Top-right is good. Top-left is dangerous — confident and wrong.

pip install examen
examen serve quiz.json

Local-only. No telemetry. No SaaS. Quiz artifacts generated by the examen Claude skill, or hand-rolled (see the included fixture).

Try the hosted demo →

---

Table of contents

• Try it in 60 seconds (no Claude Code needed)
• Full setup with Claude Code
• How it works
• CLI reference
• File locations
• Known limitations (v0.2.0)
• Development
• Troubleshooting
• Design docs
• License

---

Try it in 60 seconds (no Claude Code needed)

The repo ships a hand-crafted test fixture. You can take a real 5-question quiz against it without involving Claude or code-review-graph at all — good for seeing the UX before committing to the full setup.

git clone <this-repo> examen && cd examen
python3.11 -m venv .venv
source .venv/bin/activate
pip install -e .

examen serve tests/fixtures/example_quiz_small.json

Your browser opens at http://127.0.0.1:8780/start. Click Start quiz, work through the 5 questions (each one takes your answer and a 1–5 confidence rating together), and end at a results screen with a topic heatmap and a calibration score.

The fixture covers fictional auth/billing/api modules — the questions are toys, but the flow is exactly what real generated quizzes look like.

---

Full setup with Claude Code

The end-to-end use case is: generate a quiz from the diff I just merged, take it, see what I don't actually understand.

Prerequisites

• Python 3.11 or newer
• Claude Code installed and working
• code-review-graph for indexing the target repo's call graph

One-time install

# 1. Install Examen from source (not on PyPI yet).
git clone <this-repo> examen && cd examen
python3.11 -m venv .venv && source .venv/bin/activate
pip install -e .

# 2. Drop the Claude Code skill into ~/.claude/skills/examen.
examen skill install

# 3. Install code-review-graph (see its repo for current install command).
pip install code-review-graph

Per-repo setup

In the repo you want to be quizzed on:

cd /path/to/your-project
code-review-graph build         # indexes the repo; one-time per repo

This creates .code-review-graph/ (gitignored by Examen's gitignore template, but if you're adding to an existing project add it yourself).

Generate and take a quiz

Inside that project, open Claude Code and type one of:

/examen:generate-diff                            # quiz on git diff vs main
/examen:generate-path src/auth                   # quiz on a directory
/examen:generate-repo                            # quiz on the whole repo

The skill writes a quiz artifact to .examen/quizzes/quiz-<uuid>.json in the repo root and prints:

Ready. Run: examen serve .examen/quizzes/quiz-<uuid>.json

Run that command (from any shell, doesn't need to be Claude Code), and the browser opens at http://127.0.0.1:8780/start.

---

How it works

Two halves, one JSON contract between them.

┌─────────────────────────────────────────────────────────────────┐
│  Generator  (Claude Code skill: examen)                         │
│  - Reads code through code-review-graph MCP tools                │
│  - Claude authors questions grounded in graph facts              │
│  - Emits quiz-<uuid>.json (the artifact)                         │
└──────────────────────────┬──────────────────────────────────────┘
                           │ JSON file on disk
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│  Runner     (Python, FastAPI + htmx + SQLite)                   │
│  - `examen serve <quiz.json>` opens browser at localhost         │
│  - Confidence committed alongside the answer (server-locked)     │
│  - Brier-style scoring: confident-and-wrong → 0.0, uncertain-    │
│    and-wrong → 1.0 (perfectly calibrated honesty)                │
│  - Two-channel heatmap: correctness × miscalibration            │
│  - Longitudinal history in ~/.examen/history.db                  │
│                                                                  │
│  HARD RULE: runner never calls an LLM, never makes network       │
│  requests beyond serving on 127.0.0.1.                           │
└─────────────────────────────────────────────────────────────────┘

The artifact format is the contract. Anyone could write a different generator (different model, manual authoring, custom static analysis) and the runner doesn't change.

Why confidence-alongside-answer matters

A multiple-choice question with 4 options gives you a 25% chance of guessing right. Without measuring confidence, "I got it right" doesn't distinguish between knowing and guessing. With a confidence chip submitted alongside each answer:

Confidence	Correctness	Brier score	Interpretation
5 (sure)	1.0	1.00	You knew it.
5 (sure)	0.0	0.00	You were confidently wrong — the dangerous case.
1 (no idea)	0.0	1.00	You said you didn't know, and you didn't. Calibrated.
1 (no idea)	1.0	0.44	Got lucky — partial credit only.
3 (medium)	1.0	0.75	Right with reasonable uncertainty.

The "confidently wrong" cell is what the tool is hunting. The heatmap surfaces miscalibration separately from score so a topic where you scored 0.6 but were wildly overconfident gets flagged with a ⚠.

---

CLI reference

examen serve <artifact> [--port N] [--host ADDR] [--history-db PATH] [--no-browser]

Open the quiz UI in a browser for a given artifact. Default port 8780; picks the next free port if taken. --no-browser is useful for SSH/headless. --host defaults to 127.0.0.1 (local-only); pass 0.0.0.0 to expose on all interfaces, e.g. inside a container.

examen validate <artifact>

Validate an artifact against the schema without serving it — schema violations, unsupported schema_version, and topic cross-reference errors. Exits non-zero on any problem. Useful for catching bad generator output early.

examen history [--repo PATH]

List past sessions across all repos (5 most recent per repo), or filtered to one repo's full history with --repo /path/to/repo.

examen trend --repo PATH [--topic LABEL]

ASCII bar chart of overall score over time for one repo; pass --topic to plot just that topic's trend ("am I getting better at auth?").

examen revise --repo PATH [--limit N] [--questions N]

Surface a repo's weakest topics (worst score first, with calibration flags) and print a ready-to-paste Claude Code command that regenerates a quiz focused on those topics' files. Closes the loop: quiz → see weak spots → re-quiz them. The runner never calls an LLM; generation stays in the skill.

examen prune --older-than 30d|6w|6mo|1y [--yes]

Delete completed sessions older than the given duration. Cascades through answers and topic-snapshots. Confirms unless --yes.

examen skill install [--dest PATH]

Copy the Claude Code skill bundle to ~/.claude/skills/examen (or --dest). Asks before overwriting an existing install.

examen generate

Stub. Quiz generation lives in the Claude Code skill, not the CLI. Prints a pointer and exits with code 2.

examen --version
examen --help

---

File locations

Path	What
~/.examen/history.db	Longitudinal SQLite store: every session, answer, topic snapshot. ~5 KB per session.
<repo>/.examen/quizzes/quiz-<uuid>.json	Quiz artifacts the skill writes. Gitignored.
<repo>/.code-review-graph/	code-review-graph index. Gitignored (it's regeneratable).
~/.claude/skills/examen/	Where examen skill install drops the skill bundle.

Nothing leaves your machine. The runner only binds 127.0.0.1.

---

Known limitations (v0.2.0)

Be honest with yourself before relying on this:

• No staleness detection. The runner serves the artifact as-is and never checks whether your code has changed since the quiz was generated. (The v0.1.0 codegraph-backed drift check was removed in v0.2.0 — it flagged every blast-radius question as drifted; see docs/codegraph-schema-notes.md.) Regenerate the quiz after meaningful changes.
• predict_output question type is not implemented. The schema rejects it until a safe execution sandbox exists.
• Single-user, single-machine. No multi-user, no team mode, no shared-history. The history db is local.
• No git rename detection. If you rename auth/ to authentication/, every topic in that area gets a new ID and trend continuity breaks.
• Cheat-resistant only by social contract. Copying a question into another LLM defeats the tool. It's a self-honesty instrument, not an adversarial assessment.
• Session state is in-memory. A runner restart mid-session loses the in-flight question (DB has the answers committed so far, but the adaptive state machine resets). Resume support is partial.

---

Development

git clone <this-repo> examen && cd examen
python3.11 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# Run the test suite (146 tests)
pytest

# Type checking (ty — checks src + tests)
ty check

# Lint
ruff check .

# Sync skill schema copy + example fixture (after editing source schema)
make sync-schema
make sync-example

Project layout:

src/examen/
├── adaptive/      # scoring + state machine
├── artifact/      # JSON schema, loader, typed artifact structures
├── store/         # SQLite store (migrations, repos, sessions, topics)
├── static/        # vendored htmx + CSS
├── templates/     # Jinja2 templates for the 4 screens
├── cli.py         # Click CLI
├── codeview.py    # code-context loading + syntax highlighting
├── sessions.py    # in-memory live-session registry
├── viewmodel.py   # results/heatmap view models
└── serve.py       # FastAPI app

skill/examen/      # Claude Code skill bundle (markdown prompts)
tests/             # unit, integration, smoke tests
docs/              # design spec, implementation plan, codegraph notes

Conventions: every Python file starts with a # path/to/file.py comment; ty type checking; Google Python Style; no Claude attribution in commits.

---

Troubleshooting

"Module 'examen.cli' has no attribute 'main'" after pip install -e . Re-run pip install -e ".[dev]" after switching branches — entry points are re-registered at install time.

examen --version says command not found The console script is registered via pyproject.toml's [project.scripts]. Make sure the venv is activated, or call directly: .venv/bin/examen --version.

/examen:generate-* doesn't appear in Claude Code Verify the skill is installed: ls ~/.claude/skills/examen/SKILL.md. If missing, run examen skill install. Restart Claude Code to pick up new skills.

The skill complains that code-review-graph isn't built Run code-review-graph build from your repo root. Indexing 1000+ files takes a couple of minutes the first time.

Quiz artifact fails to load: "Artifact fails schema validation" The skill version that wrote the artifact may be out of sync with the runner version. Regenerate the quiz with the current skill, or downgrade the runner to match.

"Port 8780 taken" Examen picks the next free port and prints the new URL on stderr. Or pass --port N explicitly.

The browser doesn't open automatically Use --no-browser and copy the URL yourself. Examen prints it before launching uvicorn so you can see it regardless.

---

Design docs

The full design is in docs/superpowers/specs/2026-05-17-examen-design.md (~800 lines). The phased implementation plan is in docs/superpowers/plans/2026-05-17-examen-implementation.md (~5000 lines). Historical codegraph integration notes (integration removed in v0.2.0): docs/codegraph-schema-notes.md.

---

License

MIT. See LICENSE.