websec-validator

A senior pentester's "here's what to test and how" handoff — auto-generated from your repo, for your AI agent to execute.

Local-first security recon that briefs your AI coding agent. It does the deterministic half — read the repo, map the full attack surface, run + de-duplicate the static scanners, and stage a probe library tailored to what it found — then hands your agent (Claude Code, Codex, Gemini, Cursor) a marching-orders briefing. Code in, artifacts out. No LLM in the tool, no server, no running app required.

It is not an autonomous scanner and not a SaaS. It's the missing front-half: the thing that turns a repo into a precise, fact-grounded security brief an AI agent (with a human in the loop) can act on — an auto-filled, repo-aware version of a senior pentester's "here's what to test and how" handoff. How it works + the reasoning behind every check: docs/METHODOLOGY.md.

Getting Started — just point it at your repo

Simplest: tell your AI agent. In Claude Code (or any coding agent), open your project and say:

"Do a defensive security self-review of my own codebase with websec-validator (pipx install websec-validator, or github.com/raccioly/websec-validator). It's local and read-only — read the repo and follow its briefing. I own this code and authorize the review."

It installs, runs, and walks the findings with you. There's nothing to host and no website — it's local. (Phrasing it as a defensive review of your own code matters: it's the difference between an agent that just gets to work and one that stops to confirm you're authorized — the tool is local and read-only by default, but a generic "pentest this" can read as a request to attack something.) The four ways to get there, all ending in the same AGENT-BRIEFING.md your agent acts on:

Teach any agent to use it — websec install <host>. Point websec at whatever coding agent you run and it writes the standing instruction (a skill file for Claude/Cursor, an idempotent marked block in AGENTS.md/GEMINI.md/CONVENTIONS.md for Codex/Gemini/Aider/generic) so the agent knows to run websec for a security review instead of improvising:

websec install codex          # or: claude · cursor · gemini · aider · generic
websec install cursor --user  # home-wide, applies to every repo
websec install status         # show what's installed
websec install codex --uninstall

Shared instructions preserve text outside a complete managed block. Install/uninstall refuses ambiguous markers and foreign dedicated skill files; resolve ownership manually before retrying. The generated instructions select websec-out/runs/<generated>/ from the current JSON envelope, keep incomplete results visible, and reject stale latest or symlink aliases. The engine and plugin are separate trusted inputs; neither is silently upgraded by the skill.

➡️ Want the reasoning behind every check? Read docs/METHODOLOGY.md — what each test does and why.

Install

pipx install websec-validator   # from PyPI
brew install noir               # OWASP Noir — the route engine (50+ frameworks); regex fallback if absent
websec --version

Until the first PyPI release publishes (or for bleeding-edge), install straight from source instead: pipx install git+https://github.com/raccioly/websec-validator (or from a clone: pipx install .).

Requires Python 3.11+ (on stock macOS, python3 is often 3.9 — use pipx, which picks a newer interpreter, or install via Homebrew/pyenv). Zero Python runtime dependencies: it shells out to scanners (Trivy, OSV-Scanner, Gitleaks, Semgrep/OpenGrep, Checkov, Prowler) and Noir when present, plus per-language SAST auto-selected by stack — Bandit (Python), gosec (Go), Brakeman (Rails) — each fired only when its language is detected. It reports what's missing and never hard-fails if a tool is absent.

Or run via Docker (everything bundled, zero install)

No need to install Noir or any scanner — the image bundles them all (arch-aware, amd64 + arm64):

docker build -t websec-validator .
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD:/scan" websec-validator run /scan --out /scan/websec-out

The image carries Noir + Trivy + Gitleaks + Semgrep + Checkov; mount your repo at /scan and the artifacts land in /scan/websec-out.

Usage

websec run ./my-app                    # ← the one command: recon + stage tailored probes + emit the briefing
websec ./my-app                        # same thing — a bare path defaults to `run`
websec run ./my-app --scan             # …and also execute the available static scanners
websec run ./my-app --scan --sbom      # …and emit a CycloneDX SBOM (sbom.cdx.json) for CI/compliance
websec run ./my-app --format sarif     # SARIF 2.1.0 to stdout (for piping into CI); also always written to the run dir
websec run ./my-app --fail-on high     # exit 1 if any HIGH+ finding remains (a CI gate)
websec run ./my-app --diff main        # PR mode: scope to changed files + exact hunk line ranges
websec run ./my-app --scan --verify-secrets   # opt in to TruffleHog LIVE secret verification (egress!)
websec doctor ./my-app                 # (optional) which scanners are installed?
websec emit-context ./my-app           # print recon as a Claude Code SessionStart context envelope
websec mcp                             # run as an MCP server over stdio (typed recon tools for any MCP client)

Then point your agent at the output: "Read websec-out/AGENT-BRIEFING.md and follow it."

Prime any agent before it writes a line — the deterministic pre-brief layer

Other AI security tools make the LLM do the review from scratch (nondeterministic, per-run cost, PR-locked). websec runs first, deterministically, and hands the agent a scoped attack-surface map — so it can sit underneath an LLM reviewer and make it sharper, cheaper, and lower-FP. websec emit-context prints a compact ## SECURITY CONTEXT block wrapped in Claude Code's SessionStart additionalContext envelope. Wire it into .claude/settings.json so every session starts pre-scoped:

{
  "hooks": {
    "SessionStart": [
      { "hooks": [ { "type": "command", "command": "websec emit-context \"$CLAUDE_PROJECT_DIR\"" } ] }
    ]
  }
}

Now the agent knows your routes, auth model, tenant boundary, unguarded write endpoints, and dangerous sinks before its first turn — no LLM, same output every run. (--markdown prints the raw block for other harnesses.)

Start with run. The optional workbench commands below expose capabilities, explicit threat-feed refresh, offline reassessment and research evaluation; dynamic performs authorized live testing. recon/proof/calibrate remain development commands.

Scoping & suppression — keep the signal about your code

Most real repos vendor an input corpus: tests/, examples/, fixtures/ — sometimes whole demo apps with planted fake credentials. Those are not your product's attack surface, and websec scopes them out of the way by default without hiding anything a real leak would need:

• Fixture/example code is auto-scoped. Test/example/fixture routes are split out of the attack surface (kept in FACTS.json under fixture_endpoints, counted in the console, not probed). Secrets found in fixture files are demoted to LOW and annotated — never dropped, because a real key pasted into a test is still a committed leak. Fixture package manifests don't drive framework detection (so a CLI that vendors an Express demo isn't misread as an Express app). Pass --include-fixtures to treat all of it as product code.
• --exclude '<glob>' (repeatable) drops a path from both recon and the static scanners (gitleaks/trivy/semgrep/checkov/osv-scanner). e.g. websec run . --exclude 'tests/**' --exclude 'examples/**'.

Two SCA engines cross-check each other. With --scan, both Trivy and OSV-Scanner run: the same CVE from both collapses to one row tagged tools: [trivy, osv-scanner] (agreement → higher confidence), while OSV's broader lockfile coverage catches CVEs Trivy misses. Every CVE then flows through the reachability + EPSS/KEV enrichment above.

• .websec-ignore (repo root) — a committed, gitignore-style config for persistent scoping, so you don't re-type flags every run or in CI. Two kinds of line:

  # 1. path globs / category — DROP the finding entirely (it isn't your product)
  tests/
  examples/**
  category:supply-chain
  
  # 2. fingerprint acknowledgement — KEEP the finding, shown but NOT gating.
  #    A reason is REQUIRED (a bare fingerprint line is ignored). The fingerprint
  #    is the stable id from findings-ledger.json.
  fingerprint:b9c7f23e49bdab85  # confirmed FP: this is the scanner's own detection pattern
  

  Path/category lines remove noise that isn't your code. Fingerprint acks are the clean home for "this specific finding is a confirmed false positive" — the finding stays in the report (section 1a. Acknowledged) and in SARIF (as a suppressed result, so GitHub keeps it visible + attributable), but it's excluded from the gating total and won't trip --fail-on. Every suppression stays auditable; nothing disappears silently.

When most of a run's findings land in test/example code and there's no .websec-ignore yet, websec prints a one-line pointer to this section — it never edits your repo for you.

Prioritization — reachable × exploitable (no cloud, no LLM)

A dependency CVE list is noise until you know which ones matter. With --scan, websec enriches every dependency vulnerability along two deterministic, offline axes — the same "reachable AND exploitable" signal commercial tools sell, minus the cloud:

• Reachability — is the vulnerable package actually imported in your first-party source, or just sitting in the lockfile? websec greps the real import/require/from sites and tags each CVE imported vs declared-only (likely unreachable — verify). Declared-only vulns are the industry's #1 noise class (Snyk/Endor/Semgrep all converge here). Name-based and honest — a full call graph is out of model; unparsed ecosystems (Go, Rust…) are tagged n/a rather than guessed.
• Exploitability — each CVE is joined against a local cache of FIRST.org EPSS (exploit-probability) + CISA KEV (known exploitation reports). These are prioritization inputs alongside application context. websec intel refresh explicitly downloads public feeds without sending project data. Snapshots record source hashes/dates and freshness; consumers remain offline. Legacy flat caches are unverified.

Both are strictly additive: they annotate and re-rank, but never change a finding's severity, drop one, or add one — so they can only sharpen triage, never reintroduce a false positive.

Aim the pentest instead of competing with it

Dynamic scanners re-discover at runtime, expensively, much of what is already visible in the source. websec knows it statically, so it aims them:

• §4b — what a DAST will report, before you run one. Each static finding is mapped to the concrete scanner signature it produces (missing-csp → ZAP 10038/10055, sqli → ZAP 40018 / sqlmap, ssrf → Nuclei OAST…) with the source location that causes it. Fix those and the scan comes back clean on them — the run becomes confirmation, not discovery.
• §4b also lists the blind spots. BOLA, missing-auth, mass-assignment, RLS gaps, JWT alg-confusion, committed secrets, supply-chain — with why no scanner finds them. A clean scan is not "safe", and saying so is the point.
• §5b — a phased, pre-aimed runbook. Phase 1 safe recon → Phase 2 authz (two identities, the scanner-blind class) → Phase 3 injection, fired only at sink-backed endpoints and gated behind an explicit authorization warning. Every item names a real target and a confirm/disconfirm oracle.
• Close the loop: websec calibrate --ingest-dast zap.json --ledger findings-ledger.json feeds a real scan back — confirming or refuting each prediction — so P(real) personalizes to your app over time. Blind-spot classes are never scored: a scanner's silence there means nothing.

What it extracts (22 deterministic extractors, no LLM)

Plus derived targeting — IDOR / SSRF / open-redirect / upload / write / auth-endpoint candidates — so probes get pointed at the exact endpoints, not fired blindly.

What you get (websec-out/)

The flow

🔧 websec (deterministic)              🤖 your agent + 🧑 you
─────────────────────────────────      ─────────────────────────────────
1. recon → full attack surface     →   confirm the tenant boundary + auth model
2. run + de-dup static scanners    →   triage real-vs-noise
3. stage tailored probes           →   fill placeholders, run vs a TEST instance
4. emit AGENT-BRIEFING.md           →   propose fixes, re-run to confirm, report back

Static recon + briefing need only the code. Running the probes needs a live test instance + test credentials (the human supplies them) — the tool itself never touches a running app.

CI / enterprise integration

The recon is the same either way — these just make the output consumable by pipelines, dashboards, and non-Claude agents. All stdlib, no new dependency.

SARIF → GitHub Code Scanning. Every run writes results.sarif (SARIF 2.1.0). Upload it and each finding lands inline on the PR diff and in the Security tab, ranked by a security-severity band, with its CWE/ASVS/OWASP citation and remediation.

Gate the build. --fail-on {critical,high,medium,low} exits 1 for matching findings and 2 when requested execution is incomplete. --require-complete gates execution without a severity threshold. Extractor failures, source read loss/caps, invalid scanner reports, timeouts, and explicitly selected missing scanners remain visible in coverage.json; partial artifacts are preserved. Optional missing unselected scanners are reported as unavailable. --scanners requires --scan.

Every attempt receives a unique directory under websec-out/runs/. The atomic latest pointer advances only after a completed execution has written its artifacts; a partial attempt retains its own directory and leaves the previous completed scan selected. Completed execution describes the requested checks, not complete protection against vulnerabilities.

Reuse specialist analysis (0.14.0). Import existing SARIF without executing the analyzer or loading any source path/URL referenced by its report:

websec run ./my-app --sarif ./codeql.sarif --sarif ./other-analysis.sarif --require-complete
websec run ./my-python-app --scan --scanners bandit --fail-on high

SARIF 2.1.0 import preserves native tool/rule identities, traces and report hashes in sarif-imports.json and the unified ledger. Safe relative paths are supported; unknown analyzer execution, malformed references and caps are visible incomplete checks. Native suppressions remain metadata until locally acknowledged. Source freshness stays unverified, and imported absence cannot verify a repair. Unique producer fingerprints remain stable; collisions are deliberately report-bound.

Bandit is an optional preinstalled executable. The adapter uses built-in defaults, bypasses target configuration and ignores nosec; its native confidence and CWE remain separate from severity. Source-cache payloads have a 64 MiB per-context budget, with read loss visible in coverage. These budgets constrain input retention, not the entire Python process's memory usage.

Review changes against a baseline. --baseline <prior findings-ledger.json> tracks new, unchanged, changed, reopened, and no-longer-observed findings using versioned semantic identities. New, changed, and reopened findings participate in the severity gate. Disappearance means only “no longer observed”; it does not prove a repair. Expired or malformed dated acknowledgements reopen for review. Legacy fingerprints remain migration aliases; ambiguous aliases cannot hide distinct occurrences. A malformed or unreadable supplied baseline makes requested execution incomplete.

repair-plans.json binds each remediation plan to its original application, source digest, finding, and detector scope. This command validates operator-supplied positive and negative reports:

websec repair-verify --plan plan.json --record record.json --rerun findings-ledger.json --evidence-root ./evidence

Verification requires a hashed failing negative report for the original build. Reports must identify the same plan, finding, test and intended build; the rerun must be complete with unchanged review scope. The command validates artifacts offline and never executes their commands. Acceptance is evidence validation, not independent execution of tests by websec. See the validated evidence examples for exact dynamic, DAST and repair artifact fields.

GitHub Action (action.yml, 0.14.0): the action installs its own trusted checkout, passes inputs as data and uploads the exact current attempt's SARIF, including incomplete attempts. require-complete defaults to true; scanners names explicitly required tools when scan: true. Outputs expose run-directory, sarif-file and execution-complete. Keep the reviewed action checkout separate from untrusted target source. Pin a reviewed 0.14.0-or-later commit when adopting these changes; the historical v0.13.0 tag does not include them.

Audit evidence — websec attest. Projects an existing run into a per-control evidence table. It computes nothing and asserts nothing.

websec attest                      # latest run, human-readable
websec attest --format json
websec attest --format in-toto     # UNSIGNED Statement — sign it with your own key

Gaps are listed before evidence, in the data and in the output, because a table that leads with coverage invites absence to read as satisfaction. On a typical run, 5 of 12 rows have no websec evidence at all — most of these controls are organisational and the artifact says so.

It renders no verdict, score, percentage or badge, and the word "compliant" does not appear in any output format (there is a test). Compliance is an attribute of an assessed entity, determined by a qualified assessor and evidenced by their report — PCI SSC FAQ 1258: "no single product can provide PCI DSS compliance". Three things are declared non-goals in the artifact itself: approver independence (it lives in the forge's approval record), rollback, and PCI 6.4.2 runtime protection. The output also states that the local gate is bypassable and that its bypass record is not exhaustive.

Citations are exact, because the obvious ones are wrong. EU DORA change management is Commission Delegated Regulation (EU) 2024/1774 Art. 17 (the RTS under DORA Art. 9(4)(e)); DORA Art. 17 itself is incident management. There is no SOX article for ITGC — the domains come from SEC Release 33-8810 §II.A.2.d. PCI DSS 6.2.3 permits automated review, and 6.2.3.1 is conditional on choosing manual review, so websec is deliberately not offered against it.

The in-toto Statement is unsigned by design: a websec-signed attestation would only attest that websec ran. Sign it with your own key and identity and it becomes verifiable.

Dependency existence — websec run --network (opt-in). Offline checks cannot tell whether a declared package actually exists, which is the AI-hallucinated-dependency surface.

websec run . --network-dry-run    # print the exact names that WOULD be sent; sends nothing
websec run . --network            # HEAD registry.npmjs.org / pypi.org; ~2.8s for 52 deps

Only bare package names are sent — never versions, paths or repository identity. Suppression is applied offline, before any request: names this repo publishes (via the real workspace graph), scopes bound to a private registry in .npmrc/.yarnrc.yml, and pip index-url overrides. A private name that reaches a public registry cannot be un-sent, and a 404 on an internal name tells an attacker which name to squat.

A 404 is split by offline lockfile evidence: resolved + integrity means the package once existed, so it is dependency-unpublished-or-removed (a package pulled for malware looks exactly like this) rather than dependency-nonexistent. Findings are MEDIUM/LOW-confidence and do not fail --fail-on unless you also pass --fail-on-network — the UNKNOWN rate is outside your control, so gating on it would make registry uptime a dependency of shipping. UNKNOWN is recorded as a coverage gap, never as clean.

A 200 is not evidence of safety. A squatter who has already registered the hallucinated name also returns 200 — that is the successful attack, not the clean case.

Agent-loop gate — websec gate and websec hooks install --agent. Scanning at the merge request makes a finding a backlog item; scanning on the edit that caused it makes it a retry the agent fixes immediately.

websec gate                          # the files you just changed, ~0.3s, exit 1 if blocking
websec gate --fail-on high           # default is medium (see below)
websec hooks install --agent         # PostToolUse hook: runs the gate on every file the agent writes
websec hooks status --agent
websec hooks uninstall --agent

gate scopes the analysis, not just the report: --only narrows what is read and matched (~13x faster than a full pass), while the tree is still walked in full so stack detection, ignore policy and fixture classification are unchanged. Its default scope is the working tree — tracked modifications plus untracked files — because agent edits are uncommitted and a three-dot base...HEAD diff would see nothing. It writes no artifacts, publishes no run directory and never advances an accepted baseline.

The default threshold is medium, not high: command injection and SSRF on agent-written code are frequently rated MEDIUM, so a HIGH default would miss the main case. --min-confidence filters low-confidence leads for teams that measure them as noisy; there is no confidence floor by default, because in the loop a false block costs one turn while a miss ships.

Three deliberate properties of the hook:

• It fails open, loudly. If the gate cannot run, the hook exits 0 and says on stderr that the edit was not security-checked. A check that blocks every edit when broken gets uninstalled, and then there is no check at all. A pass means "nothing blocking was found", never "this was verified".
• It does not honour WEBSEC_SKIP_HOOK, unlike the git guardrail below. The agent can set an environment variable in a Bash call, so an env escape hatch here would be one the agent operates.
• It is developer ergonomics, not a compliance control. Settings files are editable by the user and, in a repository the agent can write to, by the agent. Only managed policy settings deployed through device management are genuinely unbypassable. This hook catches mistakes early; it cannot evidence that it ran for every change, and should not be presented as though it could. For enforcement, run websec run --fail-on … as a required status check.

A scoped pass is a fast retry signal, not a review: it does not consult cross-file evidence outside the scope, and a clean result does not mean the repository is clean. Keep running the full pass.

Local guardrail — websec hooks. Advisory and gating runs have separate responsibilities:

websec hooks install              # advisory post-commit; prints outcome and current artifact path
websec hooks install --pre-push   # severity gate (default high) plus complete-execution check
websec hooks status
websec hooks uninstall

A successful pre-push gate accepts a ledger under its severity/scanner policy. Advisory runs and failed gates never advance that accepted baseline. The first gate, or a changed policy, checks all current findings; later gates compare against the accepted ledger. Retrying an unchanged failed gate continues to fail. WEBSEC_HOOK_FAIL_ON changes the threshold; WEBSEC_HOOK_SCAN=1 executes scanners, and WEBSEC_HOOK_SCANNERS selects required adapters. The default is recon-only.

Hooks launch isolated Python bound to the trusted installed package path, including editable source installs. They preserve existing shell hooks and refuse automatic insertion into non-shell hooks. WEBSEC_SKIP_HOOK=1 is an explicit one-off override. Native hooks inspect the current working tree, not immutable snapshots of every pushed Git object. A missing runtime or concurrent incomplete gate blocks pre-push; post-commit remains advisory.

MCP server (any agent, not just Claude Code). websec mcp speaks the Model Context Protocol over stdio, exposing typed tools — websec_recon, websec_findings, websec_sarif, websec_briefing — so Cursor / Cline / Windsurf / Zed can call recon directly instead of shelling out and parsing stdout. Register it in your MCP client:

{ "mcpServers": { "websec": { "command": "websec", "args": ["mcp"] } } }

The optional HTTP transport is loopback-only and requires WEBSEC_MCP_TOKEN at startup:

# Supply WEBSEC_MCP_TOKEN through your local secret environment.
websec mcp --http --allow-root /absolute/path/to/project

Repeat --allow-root for additional projects; otherwise the startup directory is the allowed root. JSON-RPC requests require the bearer token. Host/Origin validation, framing and body limits, bounded workers, and an absolute receive deadline constrain the HTTP boundary. GET /health remains a minimal liveness endpoint. Non-loopback binding is refused. Stdio uses the launching process's trust and filesystem permissions.

Blast-radius from a knowledge graph (opt-in, zero-dep). If your repo has a graphify graph at graphify-out/graph.json (or you pass --graph <file>), websec tags each finding with how much of the app transitively depends on the vulnerable code — so a SQLi in a leaf handler and the same SQLi in a helper imported by 40 modules stop looking equally urgent:

websec run . --scan     # auto-detects graphify-out/graph.json if present

Each mapped finding gains a graph block (blast_radius, a dependents sample, community) in findings-ledger.json, and the ledger a graph_enrichment summary. It reads the graph as plain JSON — it never imports tree-sitter, so websec stays stdlib-only, zero runtime deps — and a missing implicit graph is optional. Invalid implicit graphs produce a visible diagnostic; an explicitly requested malformed, unreadable, excluded, or oversized graph makes requested execution incomplete.

Versioned contract. FACTS.json, findings-ledger.json, and findings.envelope.json all carry a schema_version: "2.0"; JSON Schemas ship in the package (schemas/facts.schema.json, schemas/ledger.schema.json, schemas/coverage.schema.json). Coverage is embedded in facts, ledger, and envelope, with SARIF execution status reflecting partial checks. Fingerprint V2 preserves V1 migration aliases. SARIF lifecycle states use its legal enum and retain richer lifecycle metadata in properties. .websec-ignore is read from the selected target by default, through the same contained read policy; an unrelated working directory no longer contributes implicit suppressions.

Named profiles and the research workbench

websec capabilities                                  # offline profile/check/limitation matrix
websec demo                                          # real scan of a bundled sample; writes nothing here
websec explain bola                                  # what a class means + how to confirm it
websec explain --list                                # every attack class this build cites
websec feedback --fingerprint <fp> --verdict false-positive \
  --reason "behind an nginx auth_request"            # report a wrong detector; metadata-only, sends nothing
websec intel status                                  # offline freshness and provenance
websec intel refresh                                 # explicit public FIRST/CISA downloads
websec intel reassess --ledger prior-ledger.json --out reassessment.json
websec research catalog                             # discover shipped suites without evaluating
websec research evaluate --suite control-scope --out suite-evaluation.json
websec research example --out proposal.json           # current detector-bound synthetic example
websec research evaluate --proposal proposal.json --out evaluation.json

--out writes a new JSON file and refuses existing files. Intel commands accept --cache-dir; status and reassessment never fetch automatically. Reassessment updates known CVEs while preserving source evidence and finding identities. Newly exploited or materially higher-risk acknowledged CVEs can reopen for review. A detector change requests a source rescan; stale/unavailable intelligence remains explicit and reassessment exits 2. Discovering new dependency CVEs requires a fresh inventory and advisory scan; unchanged-source reassessment is not such a scan.

The capability matrix exposes nine bounded profiles:

Nearest manifest boundaries identify services and keep their route/datastore evidence separate. Coverage records each selected check as completed, unknown or manual, with examined counts and limitations. Invalid analyzed configuration is an execution error. A language label or a completed no-match check does not mean comprehensive support or a secure service.

Research proposals are data-only, provenance-described candidates evaluated against an allowlist of shipped detectors. Source snippets become disposable text fixtures and are never executed. Results separate development/holdout TP/FN/TN/FP/unknown counts. Passing means eligible for human review, not automatic detector promotion. The authored synthetic holdout is regression evidence, not a statistically independent production benchmark or proof of competitor superiority.

Proof harness

websec proof clones a vuln-app corpus (VAmPI, NodeGoat, DVGA) and scores whether recon surfaces each app's documented attack surface — a deterministic, CI-trackable proxy (historical 10/10, not rerun for this detector revision). The real kill-criterion (does the briefing lift an agent's bug-finding vs a generic prompt?) is the manual A/B in corpus/PROOF-PROTOCOL.md. Full methodology, calibrated precision numbers, and the competitor-comparison protocol: BENCHMARKS.md.

Calibrated confidence

websec calibrate runs the ledger against the labeled corpus, measures how often each (attack-class, confidence) bucket is a real documented vuln, and writes calibration.json (shipped + applied at runtime). Each finding then carries P(real) with a 95% Wilson confidence interval and the sample size n — so "MEDIUM" stops being a vibe and becomes "real ~57% of the time on the historical corpus (CI 43–70%, n=51)". Unmatched findings now remain unknown, unless explicit negative evidence labels them false. Historical measurements using unmatched-as-false labels are not a fresh measurement of this detector revision. Honest caveats: the corpus is deliberately vulnerable, so the rates skew optimistic for clean production code, and small samples mean wide intervals — the CI is the headline, not the point estimate, and both tighten as the corpus grows. With thin data a bucket falls back to the per-label aggregate, then to a clearly-flagged uncalibrated prior. No ML, no deps — binomial proportion + Wilson interval; the structure upgrades to isotonic regression if a large labeled set ever exists.

Evidence controls learning. Dynamic status codes and scanner silence are observations, not truth labels. Only evidence-backed labels are admitted to the local calibration overlay; legacy unproven samples are quarantined. Unknown-only input reports no successful measurement and leaves existing fitted calibration unchanged. A DAST hit may confirm a scoped lead; absence from a report cannot refute it. Manual labels should include reviewer/evidence provenance, not only a boolean.

Dynamic phase (v2 — read-only so far)

When you have a running TEST instance, websec dynamic mints role tokens and runs the probes the static recon pointed at. v1 is read-only: authenticated cross-tenant BOLA on the group-scoped GET endpoints recon discovered.

cp dynamic-config.example.json dynamic-config.json    # TEST target + role creds (gitignored)
websec run ./my-app                                    # static recon → websec-out/latest/FACTS.json
websec dynamic --config dynamic-config.json --facts websec-out/FACTS.json
# → "14/14 cross-tenant GET reads blocked — all isolated"   (or 🚨 LEAK with the exact endpoint)

Never point it at production. Write-verb BOLA, JWT/auth attacks, and a ZAP/Nuclei two-role diff are the next dynamic probes (explicitly gated — they mutate).

Validated on

A production Next.js app, a large Express/AWS monorepo, and the VAmPI / NodeGoat / DVGA vuln-app corpus — independently reproducing a hand-done pentest's findings (tenant boundary, SSRF, file upload, cross-tenant BOLA, role/authz gaps).

Tests

python3 -m unittest discover -s tests    # stdlib only; synthetic/local-loopback tests, no public network

Releasing (maintainer)

The release train is PR-driven. Keep the version in pyproject.toml as the single source of truth, add the matching dated changelog and migration guidance, and validate the combined release branch. In the 0.x series, feature additions or incompatible contracts increment the minor version.

After reviewing current main and overlapping PRs, merge the approved release PR only after its required CI checks pass. The resulting main commit subject must be release: v0.15.3 for this release. release-tag.yml validates that subject against the package version, creates the matching tag/GitHub Release, and explicitly dispatches publish.yml at that tag. A version edit alone does not publish.

The publish workflow builds and installs the wheel, verifies its version and smoke test, then uses PyPI Trusted Publishing through the pypi environment. Verify the completed workflow, tag, GitHub Release and PyPI artifact before reporting publication success. Do not bypass failed checks with manual tags, and do not describe a locally built wheel as published.

The CLI and Claude plugin have separate delivery paths: the CLI uses PyPI semver releases; the plugin uses reviewed Git content and explicit marketplace refresh. Plugin manifests deliberately omit an independent version field. Check both sources when diagnosing stale instructions.

Status / roadmap

Version 0.15.3 provides 22 recon extractors, eleven optional scanner entries, nine named profiles, SARIF import/export, bounded source/query analysis, explicit coverage and lifecycle evidence, intelligence/research commands, and opt-in agent/hook/CI adoption. It adds an in-loop security gate (websec gate and a PostToolUse hook), analysis scoping with --only, an opt-in dependency existence check (--network), run attribution with graded assurance, recorded gate verdicts and hook bypasses, and an audit-evidence projection (websec attest). These are scoped review tools; manual profiles, unknown routes, dynamic behavior and unsupported syntax remain limitations. The in-loop gate is developer ergonomics, not a control, and attest reports evidence without rendering a compliance verdict.

The 0.14.0 migration guide explains schema 2.0 and gate/artifact changes. 0.15.3 is a patch over 0.15.2 and needs no migration. The remaining-work specification consolidates unresolved gaps and acceptance tests so overlapping old-base PRs do not become competing roadmaps. Existing runtime probes are opt-in; their earlier isolated results are not current deployment proofs.

Using it as a Claude Code skill / plugin

This repo is a Claude Code plugin. Install it once —

/plugin marketplace add raccioly/websec-validator
/plugin install websec-validator@websec-plugins

— and the bundled security-pass skill (skills/security-pass/SKILL.md) lets you just ask, in plain English, for a security pass: it runs websec, reads the briefing, and works the findings with you. For other agents the universal interface is unchanged: run the CLI, read AGENT-BRIEFING.md.

Install gotchas (field-tested):

• The install id is plugin@marketplace — websec-validator@websec-plugins (the marketplace name from .claude-plugin/marketplace.json), not @websec-validator (the repo).
• The plugin only delivers the instructions; the actual scanning is a separate Python CLI (websec). The skill checks the selected engine's version/source and reports a missing engine; installation or upgrades require an operator-selected trusted source.
• /plugin … only works in the terminal CLI. In the Claude app / Agent SDK (no /plugin), configure it in .claude/settings.json instead:

  {
    "extraKnownMarketplaces": {
      "websec-plugins": { "source": { "source": "github", "repo": "raccioly/websec-validator" } }
    },
    "enabledPlugins": { "websec-validator@websec-plugins": true }
  }
  

  This registers + enables the plugin but does not auto-fetch it — the first download still needs the CLI (/plugin install websec-validator@websec-plugins) once. (Project .claude/settings.json for a team; ~/.claude/settings.json for just you.)

Credits

Methodology + probe library are distilled from a real authenticated penetration-testing pass. This tool productizes that hand-written methodology into something an AI agent can run on any repo.

License

MIT © Ricardo Accioly

Latest source review and adoption

The upstream overlap review compares selected open PRs at exact heads with the working implementation. Recompare both target and PR heads before future integration, then rerun combined checks; overlapping old-base PRs are not automatically safe to apply. No PR was merged as part of that review.

Django URL discovery parses supported declarations and local include bindings without importing settings or executing target code. Unknown mounts remain route candidates with explicit uncertainty. A scoped read of pinned Linkding produced 53 candidates and one dynamic mount gap, not 53 confirmed HTTP paths. Framework metadata, native React views and browser renderers are distinguished; actual browser/HTTP hints remain review scope rather than proof of deployment. See the dated validation record for snapshot-bound results.

Opt-in pre-commit and PR/weekly workflow examples reuse the existing CLI, native post-commit/pre-push hooks and composite Action. The local example requires an explicit trusted Python environment; the hosted example requires a reviewed engine commit and keeps target code separate. The examples do not install hooks or activate a hosted schedule. Validation covers actual isolated CLI arguments and limited configuration structure, not a full pre-commit/YAML or hosted workflow lifecycle.

research catalog discovers the shipped control-scope suite; research evaluate --suite control-scope evaluates all three proposals and 24 authored cases. Aggregate and per-proposal results retain exact detector revision, partition metrics and uncertainty. An empty, regressed or revision-inconsistent suite exits 2. Eligibility means human review, not installation, independent real-project validation or measured vulnerability recall.

Assigned Python SQL query review now follows supported local assignments and branch joins to the query sink, preserving source traces and separate bound-value controls. Loops and cross-function behavior remain limited and disclosed. The final fourth-phase validation records 1082 tests across two Python versions and 15 isolated-wheel checks; historical corpus scores remain separate and do not establish full protection.