Skeptic

An independent engineering quality gate for AI-generated (and human-written) Python code. Skeptic doesn't take a change's word for it — it orchestrates Ruff, Pyright, Bandit, pytest/coverage, and pip-audit into one pass/fail verdict, with evidence attached to every failure, so an AI coding agent (or a human) can be required to satisfy it before a change counts as done.

Status

Phase 1 CLI MVP (v0.3), Phase 2's MCP server, and Phase 3 complete through milestone 8 (LLM verifier/pricing/billing, milestone 9, not built). All five deterministic adapters (Ruff, Pyright, Bandit, pytest, pip-audit) plus structural SOLID checks, complexity, Change Risk Score, and AI provenance tagging are wired and unit-tested against fixture repos; LLM-backed narration and verification are live-tested against the real Gemini API. See Plan.md for the full build plan and current milestone.

Install

pip install -e .

This installs skeptic as a console command, backed by Click. Verified against a clean python -m venv + pip install -e . with no other setup. Not yet on PyPI — install from a local clone until it is.

Prerequisites

Install and activate the target repo's own dependencies before running skeptic check. skeptic shells out to pyright (and pytest) using whatever Python environment is currently active — it does not install the target repo's dependencies for you. If they aren't installed, Pyright can't resolve most imports and will report a flood of reportMissingImports errors that have nothing to do with real type safety, and pytest will fail to collect tests at all. Since the default gate is zero-tolerance (types_max_errors: 0), this alone is enough to fail every real repo. The fix: cd into the target repo, activate its venv (or otherwise make sure its dependencies are installed in the active environment — e.g. pip install -r requirements.txt, uv sync, poetry install), then run skeptic check. Ruff, Bandit, and pip-audit don't need this — they analyze source/dependency manifests directly rather than resolving imports.

Usage

# Generate a starter config in your repo
skeptic init /path/to/your/repo

# Run the gate
skeptic check /path/to/your/repo

# Machine-readable output (for CI or an agent to parse)
skeptic check /path/to/your/repo --json

Example

Given a repo with an unused import, a real type error, and a vulnerable pinned dependency:

$ skeptic check .

Engineering Gate: FAIL

+----------------------------------------------------------------------------+
| Rule              | Status | Detail                                        |
|-------------------+--------+-----------------------------------------------|
| lint              | FAIL   | 1 lint findings (max allowed: 0)              |
| types             | FAIL   | 1 type errors (max allowed: 0)                |
| security_critical | PASS   | 0 critical security findings (max allowed: 0) |
| security_high     | PASS   | 0 high security findings (max allowed: 0)     |
| coverage          | PASS   | 88.9% coverage (min required: 0.0%)           |
| dependencies      | FAIL   | 12 critical/high CVEs (max allowed: 0)        |
+----------------------------------------------------------------------------+

lint findings:
  app.py:1  `os` imported but unused (F401)

types findings:
  app.py:14  Argument of type "Literal['not a number']" cannot be assigned to
  parameter "a" of type "int" in function "add" (reportArgumentType)
...

Exit code 1. Fix the issues (or explicitly relax skeptic.yaml) and the same command exits 0 with Engineering Gate: PASS.

Configuration

Edit skeptic.yaml in your repo root:

lint:
  max_errors: 0
types:
  max_errors: 0
security:
  max_critical: 0
  max_high: 0
coverage:
  min_percent: 80
dependencies:
  max_critical_cves: 0

# Optional, unset by default - see "Architecture findings" below. Neither
# blocks the gate until you uncomment it.
# architecture:
#   max_findings: 0
# ai_review:
#   max_risk_label: MEDIUM   # LOW | MEDIUM | HIGH

No skeptic.yaml? Defaults are strict (zero tolerance on everything) for lint/types/security/dependencies — architecture and ai_review are the two exceptions: they stay off until you explicitly configure them (see below for why).

For the full walkthrough — reading output, --json/CI integration, what each check actually does, troubleshooting — see docs/USAGE.md.

Architecture findings (SOLID + complexity) + narration

skeptic check also runs four deterministic, LLM-free structural checks — SRP, ISP, DIP (tool="solid"), and McCabe cyclomatic complexity (tool="complexity", functions over 10 flagged by default). They show up in the table output and --json (solid_findings/complexity_findings) either way, but whether they can fail the gate depends on skeptic.yaml:

architecture findings (informational - not yet gated):
  app/god_service.py:11  class 'GodService' touches 3 unrelated external
  systems (database, email, http) via: ... (SRP)

Not gated by default, deliberately — unlike lint/types/security/ dependencies, which default to zero-tolerance. These checks are new and haven't been broadly triaged the way an established linter has, so turning every existing repo's gate from PASS to FAIL the moment you upgrade would be a surprising, unrequested breaking change. Opt in explicitly:

architecture:
  max_findings: 0   # gates both solid and complexity findings together

Change Risk Score

Every skeptic check run (CLI table, --json, and MCP output) includes a composite LOW/MEDIUM/HIGH label — security + regression (test health) + architecture + complexity + test confidence, each scored 0-100 and always shown, not just the label:

Change Risk Score: MEDIUM (38.2/100 - security=40.0, regression=0.0, architecture=45.0, complexity=10.0, test_confidence=100.0)

The weights and LOW/MEDIUM/HIGH thresholds are a documented first-pass heuristic (see src/skeptic/core/risk_score.py), not an empirically calibrated model — treat the label as a prioritization signal, not a certified verdict. Gate on it explicitly if you want it to block:

ai_review:
  max_risk_label: MEDIUM   # fails the gate if the label exceeds this

AI provenance (skeptic provenance)

skeptic provenance /path/to/your/repo --since-ref HEAD~20

Estimates what share of recent commits landed while an AI agent was actively using skeptic-mcp against this repo, correlating each commit's timestamp against the local MCP call log. This is an approximation, not a precise record: skeptic-mcp's tools are read-only analysis, so the call log records when an agent called them, not which lines it edited — a commit landing within --window-minutes (default 15) of a logged call is labeled ai_generated (pure addition) or ai_modified (touched existing lines); everything else is human. No model identification, just the ratio. Requires no setup — with zero MCP history for a repo, everything is reported as human.

Plain-language narration (optional, costs an API call)

skeptic check /path/to/your/repo --narrate

Sends each SOLID finding to Gemini for a short "why this matters + how to fix it" explanation, printed under the finding. The LLM never originates a finding or changes the verdict — it only narrates one a deterministic check already produced, and if narration fails (no key, network error, rate limit) skeptic check still runs and reports normally, just without the narration text.

Requires GEMINI_API_KEY:

pip install -e ".[narration]"   # installs google-genai + python-dotenv
cp .env.example .env            # then fill in GEMINI_API_KEY

.env is loaded automatically (and is gitignored — never commit it). The model is gemini-3.5-flash by default, overridable via SKEPTIC_GEMINI_MODEL if it gets deprecated later — Gemini model availability shifted twice while building this feature (see src/skeptic/narration/gemini_narrator.py), so this is a real, not hypothetical, concern.

Adversarial verifier (skeptic verify)

Generates and runs attack test cases against a running instance you control — boundary values, invalid input, injection, auth bypass, IDOR, concurrency (race conditions), and failure-mode (timeout) probes — and reports pass/fail per category with the exact request that triggered each result.

pip install -e ".[verify]"   # installs google-genai + python-dotenv + httpx
# start your own app locally first, e.g.: uvicorn app.main:app --port 8000

skeptic verify /path/to/your/repo --target http://localhost:8000

Safety, by design, not as an afterthought:

• Read-only against your code. It never writes to the repo path — only generates requests and sends them to --target.
• The LLM never executes anything. Gemini returns structured data (method/path/headers/body) via a JSON schema, never code — the only thing that ever runs is an HTTP request Skeptic's own code sends. Real arbitrary-code-execution risk was a deliberate design decision not to take on for this milestone.
• Refuses non-local targets by default. --target must resolve to localhost or a private address (10.x, 172.16–31.x, 192.168.x, link- local) or the command exits immediately, before generating anything — it's sending real injection/auth-bypass/IDOR payloads, so this shouldn't be pointable at a service you don't own by accident. Pass --allow-external if you're certain the target is yours.

--diff-ref (default HEAD) focuses attack-case generation on your uncommitted changes if path is a git repo; falls back to general-purpose REST-API cases otherwise (not a git repo, or the ref doesn't exist) — never a hard failure. passed: null on a result means the heuristic genuinely can't tell (e.g. every concurrent request to a mutating endpoint succeeding identically — could be a race condition, could be a correctly-idempotent endpoint) and a human should look at detail; it's never silently coerced to a pass.

Uses the same GEMINI_API_KEY/.env as --narrate above.

MCP server (Claude Code / Cursor)

Skeptic's engine is also exposed as an MCP server, so an AI coding agent can call it mid-task instead of you running skeptic check by hand. Same engine, same adapters, same gate — the CLI and the MCP server are both thin clients of skeptic.core.

Install

pip install -e . (see above) also installs the skeptic-mcp console command, which starts the server over stdio.

Configure your project

Add this to your project's .mcp.json (Claude Code) or .cursor/mcp.json (Cursor) — not Skeptic's own repo, the repo you want the agent to check:

{
  "mcpServers": {
    "skeptic": {
      "command": "skeptic-mcp",
      "args": []
    }
  }
}

skeptic-mcp must resolve on PATH in whatever environment your editor launches subprocesses from — same as any other locally-installed MCP server. Skeptic's own repo ships this file too (dogfooding: Claude Code sessions working on Skeptic itself get the tools automatically).

Tools exposed

The same Prerequisites caveat applies: the target repo's own dependencies need to be installed/active in the environment the MCP server runs in, or types findings will mostly be import-resolution noise.

Call logging

Every call to any of the three tools is appended to a local, per-repo, append-only JSONL log at ~/.skeptic/mcp_logs/<repo-name>-<hash>.jsonl — timestamp, session id, tool name, args, and the full result. Nothing reads this back today; it's the seed of future evidence/provenance work, logged now because the cost of doing so later (once real usage has already happened without a record of it) is much higher.

Architecture

See Plan.md and src/skeptic/core/models.py for the language-agnostic schema. Python-specific tool wrappers live in src/skeptic/adapters/python/ — adding a new language later means adding a new adapter directory, not rewriting the core.

Development

pytest tests/