verify-everything 0.1.0

LLM-based code review tool that finds issues tests and linters miss

Vet : Verify Everything

Vet is a standalone verification tool for code changes and coding agent behavior.

It reviews git diffs, and optionally an agent's conversation history, to find issues that tests and linters often miss. Vet is optimized for use by humans, CI, and coding agents.

Installation

pip install verify-everything

Or install from source:

pip install git+https://github.com/imbue-ai/vet.git

Quickstart

Run Vet in the current repo:

vet "Implement X without breaking Y"

Compare against a base ref/commit:

vet "Refactor storage layer" --base-commit main

GitHub PRs (Actions)

Vet can run on pull requests.

Create .github/workflows/vet.yml:

name: Vet

permissions:
  contents: read
  pull-requests: write

on:
  pull_request:
    types: [opened, edited, synchronize, reopened]

jobs:
  vet:
    if: github.event.pull_request.draft == false
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install verify-everything
      - name: Run vet
        if: github.event.pull_request.head.repo.full_name == github.repository
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          VET_GOAL: |
            ${{ github.event.pull_request.title }}

            Additional context (not necessarily part of the goal):
            ${{ github.event.pull_request.body }}
        run: |
          set +e
          vet "$VET_GOAL" --quiet --output-format github \
            --base-commit "${{ github.event.pull_request.base.sha }}" \
            > "$RUNNER_TEMP/review.json"
          status=$?
          if [ "$status" -ne 0 ] && [ "$status" -ne 10 ]; then exit "$status"; fi

          jq --arg sha "${{ github.event.pull_request.head.sha }}" \
            '. + {commit_id: $sha}' "$RUNNER_TEMP/review.json" > "$RUNNER_TEMP/review-final.json"

          gh api "repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/reviews" \
            --method POST --input "$RUNNER_TEMP/review-final.json" > /dev/null || \
            gh pr comment "${{ github.event.pull_request.number }}" \
              --body "$(jq -r '[.body] + [.comments[] | "**\(.path):\(.line)**\n\n\(.body)"] | join("\n\n---\n\n")' "$RUNNER_TEMP/review-final.json")"
          exit 0

NOTE: This will not fail in CI if Vet finds an issue.

Environment variables

• ANTHROPIC_API_KEY is required for the default model configuration.

Using Vet with Coding Agents

Vet ships as an agent skill that coding agents like OpenCode and Codex can discover and use automatically. When installed, agents will proactively run vet after code changes and include conversation history for better analysis.

Install the skill

for dir in ~/.agents ~/.opencode ~/.claude ~/.codex; do
  mkdir -p "$dir/skills/vet/scripts"
  curl -fsSL https://raw.githubusercontent.com/imbue-ai/vet/main/skills/vet/SKILL.md \
    -o "$dir/skills/vet/SKILL.md"
  curl -fsSL https://raw.githubusercontent.com/imbue-ai/vet/main/skills/vet/scripts/export_opencode_session.py \
    -o "$dir/skills/vet/scripts/export_opencode_session.py"
  curl -fsSL https://raw.githubusercontent.com/imbue-ai/vet/main/skills/vet/scripts/export_codex_session.py \
    -o "$dir/skills/vet/scripts/export_codex_session.py"
  curl -fsSL https://raw.githubusercontent.com/imbue-ai/vet/main/skills/vet/scripts/export_claude_code_session.py \
    -o "$dir/skills/vet/scripts/export_claude_code_session.py"
done

This places the skill in ~/.agents/skills/vet/, ~/.opencode/skills/vet/, ~/.claude/skills/vet/, and ~/.codex/skills/vet/, so it is discovered by OpenCode, Claude Code, and Codex.

How it works

Vet snapshots the repo and diff, optionally adds a goal and agent conversation, runs LLM checks, then filters/deduplicates findings into a final list of issues.

Why Vet

• Verification for agentic workflows: "the agent said it ran tests" is not the same as "all tests ran successfully".
• CI-friendly safety net: catches classes of problems that may not be covered by existing tests.
• Bring-your-own-model: can run against hosted providers or local/self-hosted OpenAI-compatible endpoints.

Output & exit codes

• Exit code 0: no issues found
• Exit code 1: unexpected runtime error
• Exit code 2: invalid usage/configuration error
• Exit code 10: issues found

Output formats:

• text
• json
• github

Configuration

Model configuration

Vet supports custom model definitions using OpenAI-compatible endpoints via JSON config files searched in:

• $XDG_CONFIG_HOME/imbue/models.json (or ~/.config/imbue/models.json)
• models.json at your repo root

Example models.json

{
  "providers": {
    "openai": {
      "name": "OpenAI",
      "api_type": "openai_compatible",
      "base_url": "https://api.openai.com/v1",
      "api_key_env": "OPENAI_API_KEY",
      "models": {
        "gpt-4o": {
          "model_id": "gpt-4o-2024-08-06",
          "context_window": 128000,
          "max_output_tokens": 16384
        },
        "gpt-4o-mini": {
          "model_id": "gpt-4o-mini-2024-07-18",
          "context_window": 128000,
          "max_output_tokens": 16384
        },
        "o1": {
          "model_id": "o1-2024-12-17",
          "context_window": 200000,
          "max_output_tokens": 100000
        }
      }
    }
  }
}

Then:

vet "Harden error handling" --model gpt-4o-mini

Configuration profiles (TOML)

Vet supports named profiles so teams can standardize CI usage without long CLI invocations.

Profiles set defaults like model choice, enabled issue codes, output format, and thresholds.

See the example in this project.

License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0-only).