Part of the DEVIN ecosystem
Track: Verify · Nature: product
For: QA engineers, Developers
Interface: CLI

devin-qa-pack

Renamed (Oct 2026): this repository moved from Icaro0310/devin-qa-pack to Icaro0310/devin-assure. The PyPI package and console script stay devin-qa-pack; stars, issues and history are preserved by the redirect.

Unofficial community project. Not affiliated with, endorsed by, or sponsored by Cognition AI. "Devin" is a trademark of Cognition AI.

Linux · Personal Windows · Corporate Windows

Part of the awesome-devin ecosystem: the curated hub for the devin-* tools.

Flagship QA audit for Devin sessions: checks that what a session claims it delivered is backed by what its tool calls actually did — tests run, commits created, files written, work pushed, HTTP statuses returned — and prints a verdict per session: PASS / PARTIAL / UNVERIFIED.

The problem

Devin sessions end with the agent saying "tests passed", "committed a1b2c3d", "pushed to origin". Those are text claims — a model can write them whether or not the actions happened. Verifying them today means scrolling the transcript by hand or trusting the summary. Teams that adopt agent workflows need a cheap, repeatable way to answer did this session actually do what it claims?

Prior art

Transcript/session viewers (including Devin's own UI) show what happened; CI status checks verify outcomes after the fact; neither cross-checks the agent's delivery claims against recorded actions. The general idea — compare declared intent with observed behavior — is old (manifest-vs-manifest audits, git fsck, attestations like SLSA provenance). What didn't exist: applying it to an agent's claims vs. its own persisted tool-call log.

What makes it Devin-native

Devin persists every tool call in sessions.db → tool_call_state. This tool reads that table via devin-internals-spec and treats it as ground truth: a "tests passed" claim must be backed by a run/execute call that ran a test runner and completed; "committed sha" must show the hash in a call or in git log. Text-only auditors cannot do this — remove Devin's store and the check disappears.

Install

Python ≥ 3.10 required; install with uv (recommended) or pipx.

uv tool install devin-qa-pack

or with pipx (alternative):

pipx install devin-qa-pack

one-liner installer (pipx preferred, pip --user fallback):

curl -fsSL https://raw.githubusercontent.com/Icaro0310/devin-assure/main/install.sh | sh

GitHub Releases ship the wheel, sdist and CycloneDX SBOM per tag.

See it in action — run the reproducible Agent Assurance demo: one command generates a deliberately flawed agent session, then independently verifies what the agent actually did. No Devin install needed.

Usage

# audit one session (exact id or unique prefix)
devin-qa-pack audit --session <id> --sessions-db path/to/sessions.db

# audit everything, most recent first
devin-qa-pack audit --all --limit 20 --json

# aggregated report: one self-contained static HTML file (inline CSS,
# no JS, no external assets — opens offline)
devin-qa-pack report --sessions-db path/to/sessions.db --out report.html

# live audit at SessionEnd: audit ONLY the session that just ended
# (hook mode — writes a JSON side file, never touches the session)
devin-qa-pack session-end

# intent vs. coverage (QA-4): did the agent touch what the prompt named?
devin-qa-pack intent <session> --sessions-db path/to/sessions.db

# audit a foreign transcript instead of sessions.db (experimental)
devin-qa-pack audit --transcript .aider.chat.history.md
devin-qa-pack audit --transcript ~/.claude/projects/<slug>/<id>.jsonl

# audit a CLOUD session via the official Devin MCP events stream
# (needs $DEVIN_API_KEY — a cog_ service-user key or PAT)
export DEVIN_API_KEY=cog_...
devin-qa-pack audit --source mcp --session <cloud-session-id>

Real output, on the synthetic fixture from the quick start:

UNVERIFIED  sess-unverifiable  Mystery session — 1 claim(s): 0 verified, 0 disputed, 1 unverifiable
  [unverifiable] file/docs/spec.pdf: no matching tool call and working dir not on disk
PARTIAL  sess-disputed  Disputed session — 2 claim(s): 0 verified, 2 disputed, 0 unverifiable
  [disputed    ] tests/tests: no execute call matching `tests` recorded
  [disputed    ] commit/deadbee: no commit call recorded and no repo on disk to check
PASS  sess-verified  Verified session — 4 claim(s): 4 verified, 0 disputed, 0 unverifiable
  [verified    ] tests/pytest: `python -m pytest -q` completed (tc-pytest)
  [verified    ] commit/a1b2c3d: `git commit -m fix` completed (tc-commit)
  [verified    ] push/push: `git push origin main` completed (tc-push)
  [verified    ] file/src/report.html: tool call tc-write completed

Recorded as an asciicast: assets/demo.cast — asciinema play demo.cast.

The report subcommand audits all sessions (or one with --session, bounded with --limit) and writes a single deterministic HTML file: verdict counts, a per-session table and a per-claim breakdown with evidence and the source excerpt for every checked claim.

--sessions-db may be omitted. It auto-detects %APPDATA%/devin/cli/sessions.db on Windows and $XDG_DATA_HOME/devin/cli/sessions.db on Linux (default ~/.local/share/devin/cli/sessions.db). Always read-only.

Surfaces (--surface)

audit/report are surface-agnostic. Without an explicit --sessions-db the default --surface auto discovers every supported agent store — Devin CLI plus Copilot CLI ($COPILOT_DIR/$COPILOT_HOME/~/.copilot session-state), Codex CLI ($CODEX_HOME/~/.codex state db + rollout JSONL) and VS Code chat (User/workspaceStorage|globalStorage chat sessions, incl. .vscode-server). A surface whose store is absent simply contributes nothing — discovery never errors on a missing store.

devin-qa-pack audit --all                      # every detected surface
devin-qa-pack audit --surface copilot-cli --all
devin-qa-pack audit --surface auto --session vscode-chat:abc123

In provider mode session ids are namespaced surface:native_id (e.g. codex-cli:019ae…); --session accepts a bare native prefix or a surface:-scoped ref. Passing --sessions-db alone keeps the Devin fast path (bare ids, no other surfaces) — pass --surface to combine a pinned store with the rest. The qa_audit MCP tool takes the same surface argument.

No Devin installed? Try it on a synthetic fixture:

uv tool install devin-internals-spec
devin-inspect make-fixture /tmp/fx
devin-qa-pack audit --all --sessions-db /tmp/fx/cli/sessions.db

Exit codes: 0 every session PASS · 1 some session PARTIAL/UNVERIFIED · 2 audit could not run.

Intent vs. coverage (intent, QA-4)

devin-qa-pack intent <session> answers a different question: did the agent actually touch what the user asked for? It takes the session's first user message, extracts the paths/modules/repo names it references, extracts every path seen in tool_call_state payloads (writes, reads and commands all count as "touched") and reports two heuristic findings:

• possibly_missed — paths the prompt explicitly named (filename with an extension, absolute or ./-prefixed path) that no tool call ever referenced.
• scope_drift — touched paths sharing no directory, filename or repo anchor with anything the prompt named.

Statuses: aligned (nothing flagged) · flagged (at least one missed or drift path) · skipped (not computable — no user message, no path-ish references in the prompt, or no tool calls). Exit codes: 0 aligned · 1 flagged or skipped · 2 could not run. The same analysis rides along as an intent field in the session-end side file whenever the session resolves.

This is a heuristic — kept deliberately conservative: only file-grade references can be "missed" (a bare src/ mention or a slash-word like and/or never is); coverage is by path-segment and basename match, not semantics; and drift means "no prompt-named anchor", not "wrong file". A prompt naming no paths at all yields skipped rather than flagging every touched file.

SessionEnd hook (live audit)

devin-qa-pack session-end audits only the session that just ended and writes the verdict to a JSON side file — never into the transcript or any Devin store. Session resolution order:

1. --session-id <id> (exact id or unique prefix)
2. the session_id field of the hook payload JSON on stdin
3. the DEVIN_SESSION_ID env var (exported by the hook dispatcher)
4. the most recently active session in sessions.db

The side file defaults to <data-dir>/qa/<session-id>.json (same platform data dir as the store auto-detection; --data-dir and --out override) and contains {session_id, verdict, claims: [...], audited_at} — the same claim shape as audit --json, plus an optional intent field (the QA-4 prompt-vs-touched-paths analysis) when the session resolves. A one-line summary is also printed. --limit N bounds the number of claims verified.

Fail-soft: session-end always exits 0 once it ran — the verdict travels in the side file, so a hook can never fail the host session. A session that cannot be resolved produces a SKIPPED verdict (still written to the side file when an id is known). Non-zero exits are reserved for usage errors (2), consistent with the other subcommands.

Register it as a SessionEnd hook (hooks.json entry for the devin-powerups hook dispatcher — it pipes the hook payload JSON on stdin and exports DEVIN_SESSION_ID):

{
  "SessionEnd": [
    {
      "matcher": "",
      "hooks": [
        {
          "type": "command",
          "command": "devin-qa-pack session-end",
          "timeout": 30
        }
      ]
    }
  ]
}

Pre-commit gate

The repo root ships .pre-commit-hooks.yaml — reference it from a consumer's .pre-commit-config.yaml:

- repo: https://github.com/Icaro0310/devin-assure
  rev: <tag-or-sha>
  hooks: [{ id: devin-qa-pack-gate }]

Requires devin-qa-pack on PATH (pipx install devin-qa-pack or uv tool install devin-qa-pack). On every commit the gate runs devin-qa-pack audit --all --limit 1 --within . from the repo root: it reads the auto-detected sessions.db, keeps only sessions whose working_directory is this repo and audits the newest one. A PARTIAL or UNVERIFIED verdict exits 1 and blocks the commit; the gate fails closed (exit 2) when no sessions.db is reachable or no session matches this repo. Fully read-only and offline — --online is never passed. Widen it via args: (e.g. args: [--limit, "20"]); --session and --transcript cannot be selected through args: because --all stays in the entry and the scope flags are mutually exclusive — run devin-qa-pack audit --session <id> directly instead.

Auditing other agents (experimental)

audit --transcript runs the same claim/evidence model over non-Devin transcripts via the adapters in src/devin_qa_pack/adapters/:

• aider — .aider.chat.history.md. > /run and > /test records become execute calls with unknown status (the history does not store exit codes), so test claims resolve unverifiable; commit claims are checked against git log in the working directory.
• claude-code — session .jsonl transcripts. Bash → execute, Write/Edit/MultiEdit/NotebookEdit → write, tool_result blocks mark calls completed/failed.

--format overrides extension-based detection; --cwd sets the directory used for git/file checks. These adapters are experimental — formats outside the recorded subsets degrade to UNVERIFIED, never to a false PASS.

Auditing cloud sessions (Devin MCP)

audit --source mcp --session <id> audits sessions that ran in the Devin cloud — automations, API-created runs, anything in your org — using the official MCP server's devin_session_events stream (https://mcp.devin.ai/mcp, Bearer + DEVIN_API_KEY; set DEVIN_ORG_ID when the token needs explicit org context).

The events stream is the cloud analog of tool_call_state: shell_process_started carries the full command plus a parsed sub-command chain, terminal_update streams base64 stdout, and shell_process_completed carries a first-class exit_code — so claims resolve with the same verified/disputed/unverifiable semantics as local audits, plus per-event timestamps the local store lacks. Verified live against a Pro-plan PAT (see devin-internals-spec's docs/EVIDENCE-EQUIVALENCE.md for the full surface study).

Two honest degradations, by design:

• Disk/git fallbacks have no anchor — the cloud VM's starting_dir is not on your disk, so file/commit claims rely on event evidence only.
• file/mcp/git/browser event contents are mapped generically (kind from category, status from the event-type suffix); only shell shapes are verified end-to-end so far.

Works with Devin alone (Devin-only mode)

devin-qa-pack is an offline, read-only audit of recorded Devin sessions. It never calls an LLM, makes no network calls by default, and never writes to Devin's stores — a safe pick for restricted machines.

Platform support

Tested on Windows and Linux (windows-latest + ubuntu-latest in CI). The CLI session DB is auto-detected from %APPDATA%/devin/cli/sessions.db on Windows and $XDG_DATA_HOME/devin/cli/sessions.db on Linux (default ~/.local/share/devin/cli/sessions.db). A legacy ~/.config/devin layout is also checked. macOS uses ~/Library/Application Support/devin/. Pass --sessions-db to override.

--online (QA-2, opt-in) enables live HEAD checks of deploy/URL claims ("deployed to https://…"), and only against --allow-domain hosts — network is off by default and the audit is fully offline otherwise.

Limitations

• chat_message and tool_call_*_json payloads are unstable formats (see devin-internals-spec SCHEMA.md). Decoding is defensive; rows that can't be read resolve claims to unverifiable, never to disputed.
• Claim extraction is heuristic: it looks for delivery phrasing ("tests passed", "committed ", "created
• HTTP-status claims are checked only against recorded tool-call output: a claim is verified when a recorded status matches, disputed when outputs record a different status and unverifiable when no output records any status. No request is ever replayed.
• File/commit checks use sessions.working_directory only when it exists on disk and is a git repo — otherwise they rely on tool-call evidence alone.
• The intent analysis is explicitly heuristic: prompt path extraction is regex-based (bare directory mentions without extensions and extensionless a/b tokens can be missed as references), "touched" includes reads and commands not just writes, and later user follow-ups are ignored — only the first user message sets intent. Treat possibly_missed/scope_drift as review hints, not verdicts.
• The transcript adapters cover a subset of Aider/Claude Code formats: aider's history cannot prove a /run succeeded (no exit codes), and unrecognized tool names become other — still searchable evidence, but weak. Real-world transcripts that differ degrade to UNVERIFIED.
• It verifies that actions happened, not that the work is good. Green tests in a tool call don't prove the fix is correct.
• Read-only, offline; no real-time monitoring. A read-only MCP server (devin-qa-pack-mcp, mcp extra) exposes qa_audit — the same payload as audit --json; applying nothing mutating.

Development

pip install -e ".[dev]"
python -m pytest

Adapters (MCP / Devin skill / plugin)

• MCP server: pip install 'devin-qa-pack[mcp]' then run devin-qa-pack-mcp (stdio). Exposes qa_audit — the same JSON as devin-qa-pack audit --json. Read-only tools only.
• Devin plugin + skill: devin plugins install Icaro0310/devin-assure#packages/qa-pack/adapters. The manifest launches the server through uvx --from 'devin-qa-pack[mcp]' devin-qa-pack-mcp, which resolves once the next PyPI release ships. Until then, an editable install does not change what uvx --from resolves — either run the source-installed devin-qa-pack-mcp directly, or point a local manifest copy at the checkout: uvx --from './packages/qa-pack[mcp]' devin-qa-pack-mcp.

MCP host configs

Same stdio server in other hosts — server name devin-qa-pack:

Zed — context_servers in ~/.config/zed/settings.json:

{ "context_servers": { "devin-qa-pack": { "command": { "path": "devin-qa-pack-mcp", "args": [] } } } }

JetBrains AI Assistant — Settings → Tools → AI Assistant → Model Context Protocol → Add:

{ "mcpServers": { "devin-qa-pack": { "command": "devin-qa-pack-mcp" } } }

Antigravity — ~/.gemini/config/mcp_config.json:

{ "mcpServers": { "devin-qa-pack": { "command": "devin-qa-pack-mcp" } } }

When to use this

• You want to verify that a finished Devin session actually ran tests, created commits, wrote files, or pushed — not just claimed to.
• You are gating agent output in CI and need a machine-readable verdict (PASS/PARTIAL/UNVERIFIED) with exit codes.
• You want to audit many sessions at once, offline, without sending transcripts to an LLM — and optionally publish a single static HTML report (devin-qa-pack report).
• You are on a restricted machine: the tool is read-only and makes no network calls by default (the opt-in --source mcp and --online checks are off unless requested).

When NOT to use this

• You need a review of code quality or correctness — it verifies that actions happened, not that the work is good.
• You need real-time monitoring — the devin-qa-pack-mcp MCP server is read-only batch audit, not live watching (still on the M2 roadmap).
• Your agent is not Devin — the ground truth comes from Devin's sessions.db.

FAQ

How do I verify a Devin session actually ran the tests it claims? Run devin-qa-pack audit --session <id>. It reads the session's tool_call_state rows from sessions.db, extracts delivery claims like "tests passed", and checks each against recorded tool calls — a claim without matching evidence resolves to UNVERIFIED or PARTIAL, not PASS.

Does devin-qa-pack need network access or an API key? By default, no — it is a read-only audit of the local sessions.db. Two opt-in exceptions exist: audit --source mcp queries mcp.devin.ai (requires DEVIN_API_KEY, fails closed without it; DEVIN_ORG_ID is sent only when set, for tokens that need explicit org context) and --online/--allow-domain HEAD-checks allowlisted URLs. Neither runs unless requested; it never calls an LLM and never writes to Devin's stores.

Where does devin-qa-pack find sessions.db? It auto-detects %APPDATA%/devin/cli/sessions.db on Windows, $XDG_DATA_HOME/devin/cli/sessions.db on Linux (~/.local/share/devin/cli/sessions.db by default), ~/Library/Application Support/devin/ on macOS, plus a legacy ~/.config/devin layout. Override with --sessions-db.

License

MIT — see LICENSE.

---

If this saved you debugging time, a ⭐ on the repo helps others find it.