Skeptic

An independent engineering quality gate for Python code — AI-generated or human-written.

Skeptic doesn't take a change's word for it. It orchestrates a battery of established, best-in-class tools — Ruff, Pyright, Bandit, pytest/coverage, and pip-audit — into a single, evidence-backed pass/fail verdict, and attributes every finding against a baseline so a change is judged only on what it actually introduced. The result is a gate an AI coding agent, a CI pipeline, or a human reviewer can be required to satisfy before a change counts as done — with every failure traceable to the exact file, line, and rule behind it.

Table of contents

• Why Skeptic
• Installation
• Prerequisites
• Quick start
• CLI reference
• skeptic check — the quality gate
• skeptic diff — change-aware evaluation
• Configuration
• Architecture findings and the Change Risk Score
• AI provenance
• Plain-language narration
• Adversarial verifier (skeptic verify)
• MCP server
• Golden Cases (skeptic eval)
• Architecture of this repository
• Development
• License

Why Skeptic

Static analysis tools already exist. What's missing is a single, opinionated verdict that ties them together, treats a pre-existing problem differently from one a change just introduced, and hands back evidence rather than a summary you have to trust.

• One verdict, five tools. Ruff, Pyright, Bandit, pytest/coverage, and pip-audit run together and roll up into a single pass/fail gate, each rule independently reported with the exact findings behind it.
• Change-aware, not just repo-aware. skeptic diff classifies every finding as INTRODUCED, PRE_EXISTING, or RESOLVED relative to a baseline, so a change is never blocked by a problem it didn't cause.
• Structural checks, not just lint. SOLID-principle violations (SRP, ISP, DIP) and cyclomatic complexity are detected on top of the standard tool findings.
• A documented, transparent risk score. Every contributor to the Change Risk Score is listed individually — never a black-box number.
• AI-agent-ready. A first-class MCP server exposes the same engine to Claude Code, Cursor, and other MCP-compatible agents, plus commit-level provenance tagging and an optional LLM-backed adversarial verifier.
• CI-native. Deterministic exit codes and a stable --json schema for every command.

Installation

pip install skeptic-cli

Installs the skeptic console command.

Working from a local clone instead (for example, to pick up unreleased changes)? pip install -e . does the same thing from source.

Prerequisites

Install and activate the target repository's own dependencies before running skeptic check. Skeptic shells out to pyright and pytest using whichever Python environment is currently active — it does not install the target repository's dependencies for you. If they aren't installed, Pyright cannot resolve most imports and reports a flood of reportMissingImports errors unrelated to real type safety, and pytest fails to collect tests at all. Because the default gate is zero-tolerance (types.max_errors: 0), this alone is enough to fail almost any real repository.

The fix: cd into the target repository, activate its environment (or otherwise ensure its dependencies are installed in the active environment — pip install -r requirements.txt, uv sync, poetry install, etc.), then run skeptic check. Ruff, Bandit, and pip-audit don't need this — they analyze source and dependency manifests directly rather than resolving imports.

Quick start

# Generate a starter config in your repo
skeptic init /path/to/your/repo

# Run the gate
skeptic check /path/to/your/repo

# Machine-readable output, for CI or an agent to parse
skeptic check /path/to/your/repo --json

CLI reference

Every command supports --json for machine-readable output with a stable schema, alongside a formatted table/text view for interactive use.

skeptic check — the quality gate

Example

Given a repository with an unused import, a real type error, and a vulnerable pinned dependency:

$ skeptic check .

Engineering Gate: FAIL

+----------------------------------------------------------------------------+
| Rule              | Status | Detail                                        |
|-------------------+--------+-----------------------------------------------|
| lint              | FAIL   | 1 lint findings (max allowed: 0)              |
| types             | FAIL   | 1 type errors (max allowed: 0)                |
| security_critical | PASS   | 0 critical security findings (max allowed: 0) |
| security_high     | PASS   | 0 high security findings (max allowed: 0)     |
| coverage          | PASS   | 88.9% coverage (min required: 0.0%)           |
| dependencies      | FAIL   | 12 critical/high CVEs (max allowed: 0)        |
+----------------------------------------------------------------------------+

lint findings:
  app.py:1  `os` imported but unused (F401)

types findings:
  app.py:14  Argument of type "Literal['not a number']" cannot be assigned to
  parameter "a" of type "int" in function "add" (reportArgumentType)
...

Exit code 1. Resolve the findings (or explicitly relax skeptic.yaml) and the same command exits 0 with Engineering Gate: PASS — suitable for direct use in a CI job or a shell && chain.

See docs/USAGE.md for the full walkthrough: reading output, --json/CI integration, what each check does, and troubleshooting.

skeptic diff — change-aware evaluation

skeptic check answers "is this repository healthy?" skeptic diff answers a different, often more useful question: what did this specific change introduce, fix, or leave untouched?

skeptic diff                  # working tree vs HEAD
skeptic diff --base main      # working tree vs the merge-base of HEAD and main
skeptic diff HEAD~1 HEAD      # two explicit refs, no working tree involved

Every finding from every adapter is attributed against a baseline:

Attribution is identity-based (rule + file, refined to the enclosing symbol when available) and survives line drift and file renames, so a finding that merely moved doesn't read as a new one, and a pre-existing critical issue never blocks a change that didn't cause it.

Comparing HEAD~1 -> HEAD

INTRODUCED (1):
  app.py:12 [high] Possible SQL injection vector through string-based query construction (B608)

Change Risk Score: MEDIUM (52.0/100)
  +32.0 introduced HIGH security finding
  +12.0 no related test file detected

Change Verdict: BLOCK

Verdict policy:

• BLOCK — an introduced critical/high-severity finding, or newly failing tests. Exit code 1.
• WARN — an introduced medium-severity finding, or a significant test coverage gap. Exit code 0.
• PASS — otherwise.

The change is also evaluated for related-test coverage (by naming convention, honestly reported as "no related tests detected" rather than "this change is untested" — the two are not the same claim) and dependency manifest changes (added/removed/version-changed packages). The baseline (and, for a two-ref comparison, the target) is always checked out into an isolated git worktree — your working tree is never mutated.

--json emits the same underlying attribution, risk score, and verdict as the text output — never two independently computed answers.

Configuration

Edit skeptic.yaml in your repository root:

lint:
  max_errors: 0
types:
  max_errors: 0
security:
  max_critical: 0
  max_high: 0
tests:
  max_failures: 0
coverage:
  min_percent: 80
dependencies:
  max_critical_cves: 0

# Optional, unset by default — see "Architecture findings" below. Neither
# blocks the gate until you uncomment it.
# architecture:
#   max_findings: 0
# ai_review:
#   max_risk_label: MEDIUM   # LOW | MEDIUM | HIGH

With no skeptic.yaml, every threshold defaults to zero-tolerance for lint/types/security/tests/dependencies. architecture and ai_review are the two exceptions — they stay off until explicitly configured (see below).

Architecture findings and the Change Risk Score

skeptic check also runs four deterministic, tool-free structural checks — SRP, ISP, DIP (tool="solid"), and McCabe cyclomatic complexity (tool="complexity", functions over 10 flagged by default). They appear in the table output and --json (solid_findings/complexity_findings) regardless of gating:

architecture findings (informational — not gated by default):
  app/god_service.py:11  class 'GodService' touches 3 unrelated external
  systems (database, email, http) via: ... (SRP)

Unlike lint/types/security/dependencies, these are not gated by default — they're newer and haven't been broadly triaged the way an established linter has, so upgrading shouldn't silently flip an existing repository's gate from PASS to FAIL. Opt in explicitly:

architecture:
  max_findings: 0   # gates both solid and complexity findings together

Change Risk Score

Every skeptic check run (CLI table, --json, and MCP output) includes a composite LOW/MEDIUM/HIGH label — security, regression (test health), architecture, complexity, and test confidence, each scored 0–100 and always shown alongside the label, not hidden behind it:

Change Risk Score: MEDIUM (38.2/100 - security=40.0, regression=0.0, architecture=45.0, complexity=10.0, test_confidence=100.0)

The weights and thresholds are a documented, first-pass heuristic (see src/skeptic/core/risk_score.py), not an empirically calibrated model — treat the label as a prioritization signal. Gate on it explicitly if you want it to block:

ai_review:
  max_risk_label: MEDIUM   # fails the gate if the label exceeds this

AI provenance (skeptic provenance)

skeptic provenance /path/to/your/repo --since-ref HEAD~20

Estimates what share of recent commits landed while an AI agent was actively using skeptic-mcp against a repository, correlating each commit's timestamp against the local MCP call log.

This is an approximation, not a precise record: skeptic-mcp's tools are read-only analysis, so the call log records when an agent called them, not which lines it edited. A commit landing within --window-minutes (default 15) of a logged call is labeled ai_generated (pure addition) or ai_modified (touched existing lines); everything else is human. Requires no setup — with no MCP history for a repository, everything is reported as human.

Plain-language narration (optional)

skeptic check /path/to/your/repo --narrate

Sends each SOLID finding to an LLM for a short "why this matters and how to fix it" explanation, printed under the finding. The model never originates a finding or changes the verdict — it only narrates a finding a deterministic check already produced. If narration fails (missing key, network error, rate limit), skeptic check still runs and reports normally, minus the narration text.

Requires GEMINI_API_KEY:

pip install -e ".[narration]"   # installs google-genai + python-dotenv
cp .env.example .env            # then fill in GEMINI_API_KEY

.env is loaded automatically and is gitignored — never commit it. The model is configurable via SKEPTIC_GEMINI_MODEL if the default is deprecated.

Adversarial verifier (skeptic verify)

Generates and runs attack test cases against a running instance you control — boundary values, invalid input, injection, auth bypass, IDOR, concurrency (race conditions), and failure-mode (timeout) probes — and reports pass/fail per category with the exact request that triggered each result.

pip install -e ".[verify]"   # installs google-genai + python-dotenv + httpx
# start your own app locally first, e.g.: uvicorn app.main:app --port 8000

skeptic verify /path/to/your/repo --target http://localhost:8000

Safety is a design constraint, not an afterthought:

• Read-only against your code. It never writes to the repository path — only generates requests and sends them to --target.
• The model never executes anything. Attack cases are returned as structured data (method/path/headers/body) via a JSON schema, never as code — the only thing that ever runs is an HTTP request Skeptic's own code sends.
• Refuses non-local targets by default. --target must resolve to localhost or a private address (10.x, 172.16–31.x, 192.168.x, link-local), or the command exits immediately before generating anything — it sends real injection/auth-bypass/IDOR payloads, so it shouldn't be pointable at a service you don't own by accident. Pass --allow-external only against a target you're certain is yours.

--diff-ref (default HEAD) focuses attack-case generation on your uncommitted changes when path is a git repository; it falls back to general-purpose REST API cases otherwise. passed: null on a result means the heuristic genuinely can't tell (for example, every concurrent request to a mutating endpoint succeeding identically could indicate a race condition or a correctly idempotent endpoint) — a human should look at detail; it's never silently coerced to a pass.

Uses the same GEMINI_API_KEY/.env as --narrate above.

MCP server (Claude Code / Cursor)

Skeptic's engine is also exposed as an MCP server, so an AI coding agent can call it mid-task instead of a human running skeptic check by hand. Same engine, same adapters, same gate — the CLI and the MCP server are both thin clients of skeptic.core.

Install

pip install -e . (see Installation) also installs the skeptic-mcp console command, which starts the server over stdio.

Configure your project

Add this to your project's .mcp.json (Claude Code) or .cursor/mcp.json (Cursor) — the repository you want the agent to check, not Skeptic's own:

{
  "mcpServers": {
    "skeptic": {
      "command": "skeptic-mcp",
      "args": []
    }
  }
}

skeptic-mcp must resolve on PATH in whatever environment your editor launches subprocesses from, the same as any other locally installed MCP server.

Tools exposed

The same Prerequisites requirement applies: the target repository's own dependencies need to be installed and active in the environment the MCP server runs in, or types findings will mostly be import-resolution noise.

Call logging

Every call to any of the three tools is appended to a local, per-repository, append-only JSONL log at ~/.skeptic/mcp_logs/<repo-name>-<hash>.jsonl — timestamp, session ID, tool name, arguments, and the full result. This log is the basis for AI provenance tagging.

Golden Cases (skeptic eval)

Skeptic's own benchmark suite: given a repository with a known engineering failure, does Skeptic detect it? See golden_cases/manifest.yaml and each case's README.md for the fixture format and the current case set (security, architecture, complexity, dependency, testing, regression, adversarial, and change-level cases under golden_cases/changes/).

skeptic eval                          # runs golden_cases/, deterministic profile
skeptic eval --json                   # machine-readable, stable schema
skeptic eval --category security      # filter to one category
skeptic eval --compare benchmarks/v0.5.2.json   # regression check against a saved baseline
skeptic eval --changes                # also run the skeptic diff regression suite (slower, opt-in)

Adversarial cases (LLM-verifier-based, e.g. IDOR/auth-bypass/injection against a live endpoint) are authored and schema-valid but not yet executed by eval — eval reports how many are defined without spending an LLM call on them. .github/workflows/golden-cases.yml runs the deterministic profile (plus --changes) in CI and fails the build only on a regression against the latest saved baseline in benchmarks/, not on an absolute threshold.

Architecture of this repository

See src/skeptic/core/models.py for the language-agnostic finding schema. Python-specific tool wrappers live in src/skeptic/adapters/python/ — adding support for another language means adding a new adapter directory, not rewriting the core engine.

Development

pip install -e ".[dev]"
pytest tests/

Contributions are welcome. Please open an issue to discuss substantial changes before submitting a pull request.

License

MIT © Hamza Shaikh