vibeaudit
A security and quality audit for apps built with AI coding assistants, and for apps that use LLMs.
vibeaudit reads a whole repository (frontend, backend, database rules, CI config and the AI parts) and gives you a prioritised list of what is actually dangerous or broken, each with file and line, why it matters and how to fix it.
It targets the mistakes that AI-generated ("vibe-coded") apps keep shipping: API keys in the browser
bundle, API routes with no auth check, Supabase tables without Row Level Security, packages the assistant
invented, model output piped into exec, unbounded token spend and security placeholders that were never
finished. Beyond security it catches what makes generated apps break in production: unfinished stubs and
missing imports, N+1 queries and money stored as floats, requests without timeouts, buttons that do
nothing, inaccessible forms and frontend calls to API routes that do not exist.
- Works offline with no API key. The network is used only for package-registry and vulnerability
lookups (
--offlineturns that off); an LLM is used only if you ask for it. - Pure Python, no dependencies on 3.11+ (
tomlion 3.9 and 3.10). Linux, macOS and Windows. - 172 rules: 171 rules in 12 categories plus the optional LLM review, mapped to CWE, the OWASP Top 10 and the OWASP Top 10 for LLM Applications.
- Terminal output plus JSON, SARIF 2.1.0 (GitHub code scanning), Markdown, HTML, an SVG badge and a fix brief you can paste into an AI assistant.
- Fits into CI (GitHub Action,
--diff), pre-commit (--staged) and coding agents (an MCP server).
Contents: Quickstart - Architecture - Example output - What it checks - Install - Commands - Automatic fixes - Configuration - CI and pre-commit - Editors - MCP server - Scoring - False positives - Limits - Accuracy & Benchmarks - Contributing
Architecture
flowchart LR
Repo["Target Repository (Frontend, Backend, DB Rules, CI, AI/MCP)"] --> Discovery["Stack & Route Discovery (Flask, FastAPI, Next.js, Express, Supabase)"]
Discovery --> PyAST["Python AST & Taint Engine"]
Discovery --> JSLex["Offset-Preserving JS/TS Lexer"]
Discovery --> CfgScan["Config, RLS, Supply-Chain & OSV Scanner"]
Discovery --> ExtTools["Optional Tool Merger (gitleaks, bandit, semgrep, osv-scanner)"]
PyAST & JSLex & CfgScan & ExtTools --> Rules["172 Rules Across 12 Domains (CWE & OWASP LLM Mapped)"]
Rules --> Dedup["Deduplication, Domain Scoring & Fix Generator"]
Dedup --> Out["Outputs: Terminal, SARIF 2.1.0, HTML, Studio UI, LSP & MCP Server"]
Quickstart
pip install vibeaudit
vibeaudit scan path/to/your/project
vibeaudit scan path/to/your/project --report html
The first command installs it, the second prints findings in the terminal, the third also writes
vibeaudit-report.html, a self-contained page you can open in a browser. The exit code is 1 when there is
a finding at high or above, so the same command works as a CI gate.
Example output
$ vibeaudit scan tests/fixtures/vuln_flask_llm
Stack: Flask, OpenAI SDK
[CRITICAL] AUTH008 Auth check stub always succeeds
app.py:24 `verify_token()` always returns True.
> def verify_token(token):
Why: A function that is supposed to verify a token, password or permission just returns true...
Fix: Implement the real check (verify signature, expiry, user and role) or use a maintained auth library.
[CRITICAL] BE002 Command injection
app.py:50 Request data flows into a shell command.
> os.system("echo " + request.args.get("name", ""))
...
Summary
critical: 3 high: 20 medium: 15 low: 7 total: 45
...
Score: 0/100 Grade: F
Domains: Security 0 AI 42 Quality 93 Data 96 Reliability 99 UI 100
That project is one of the deliberately vulnerable sample apps in tests/fixtures/.
What it checks
| Category | What it finds |
|---|---|
| Secrets (SEC) | provider keys and tokens, private keys, hard-coded or default signing secrets, database URLs with passwords, committed or un-ignored .env files, secrets in NEXT_PUBLIC_/VITE_ variables, a Supabase service_role key outside the server, secrets in logs, keys baked into committed bundles; in preview: other high-entropy secrets, hard-coded passwords |
| Supply chain (DEP) | packages that do not exist (hallucinated names), typosquats of popular packages, packages first published in the last 30 days, known vulnerabilities (OSV) for pinned versions, missing lockfiles, install scripts that run remote code, unpinned MCP servers; in preview: extra package indexes (dependency confusion), vulnerable transitive dependencies |
| Backend (BE) | SQL and NoSQL injection, SSRF, path traversal, server-side template injection, mass assignment, debug mode, permissive CORS, disabled TLS verification, weak password hashing, insecure randomness, unverified JWTs, leaked error details; in preview: command injection, eval/exec, unsafe deserialization and open redirects (these four still gate), no server-side validation, payment amounts taken from the client, email header injection |
| Frontend (FE) | LLM APIs called from the browser, authorization decisions in browser storage, postMessage without origin checks, tokens in localStorage, production source maps; in preview: XSS through raw HTML sinks (still gates), hard-coded localhost URLs, server actions that expose a server secret |
| Auth (AUTH) | endpoints that change data or spend money without authentication, admin routes without a role check, IDOR-shaped handlers, unverified webhooks, tables without Row Level Security, permissive policies, open Firebase rules, auth functions that always return true, CSRF disabled, login without rate limiting; in preview: JWTs accepted without pinned algorithms, session cookies without SameSite or CSRF protection, webhook signatures without replay protection, rate limits keyed on X-Forwarded-For, password changes that keep old sessions, queries not scoped to the tenant, admin or feature-flag endpoints without a role check |
| Config (CFG) | containers running as root, secrets in images, GitHub Actions script injection, pull_request_target checking out untrusted code, any Host header accepted, default database passwords, missing security headers, .env copied into an image, third-party GitHub Actions not pinned to a commit, unencrypted RDS and EBS storage (Terraform and CloudFormation); in preview: unpinned base images, permissions: write-all, privileged or host-mounted Kubernetes containers and missing resource limits, public S3 buckets, admin and database ports open to the internet, secret files or build arguments in the final image, pull request code run with secrets |
| LLM and agents (LLM) | untrusted input in the system prompt, model output in a dangerous sink, no token limit, LLM endpoints without rate limiting, agent tools with dangerous capabilities, dangerous agent options, system prompts shipped to the browser, unsafe model loading, unbounded agent loops, vector search without a tenant filter, model output parsed as JSON without error handling, retired model ids, MCP servers with whole-disk access or plain HTTP, secrets in MCP configs, agents that read untrusted content and also have dangerous tools; in preview: coding agents configured to act without asking, streamed completions not cancelled on disconnect, prompts and completions written to logs, model output as a command argument, agent tool loops without a step limit, user text in a prompt without delimiters, retrieved documents stored without cleaning, no per-user cost limit, LLM code without tests or evals, prompt-injection text in an agent instruction file |
| Code completeness (QA) | no CI, TypeScript checking disabled, the same function copied into several files; in preview: no tests, swallowed exceptions, security TODOs, unfinished stubs, imports of files or names that do not exist, packages imported but not declared, Pages Router code in an App Router file, made-up data used as a fallback |
| Data (DATA) | whole user records returned to the client, destructive migrations, money as float, private tables readable by everyone, timestamps without time zone, a database client per request, SQLite on a serverless host; in preview: N+1 queries, env vars missing from the example env file, several writes without a transaction, indexes built without CONCURRENTLY, foreign keys without an index, Redis keys without an expiry, personal data in vector store metadata, cache stampedes, background jobs that are not safe to retry |
| Reliability (REL) | HTTP requests without a timeout, blocking calls in async handlers, webhooks that are not idempotent, uploads without a size limit, in-memory state on serverless hosts, no health-check endpoint |
| UI, UX and accessibility (UI) | images without alt text, click handlers the keyboard cannot reach, unlabeled form fields, controls that do nothing, index keys on changing lists, hydration mismatches, effects without cleanup, lorem ipsum, no error page, missing metadata or viewport, links to pages that do not exist; in preview: deletes without confirmation, no loading or error state, images without width and height, form errors that screen readers do not hear, focus outlines removed without a replacement, links to assets that are not in the public folder |
| Frontend-backend contracts (API) | frontend calls to API paths no route handles, and calls with a method the route does not accept |
| Go, Java/Kotlin and C# (GO, JAVA, CSHARP), all in preview | goroutines started by a handler that never stop (GO001), response bodies never closed (GO002), maps written concurrently without a lock (GO003), SQL built with formatting (GO004); Spring controllers with no security configuration (JAVA001), exposed Actuator endpoints (JAVA002), a security chain that permits every request (JAVA003), polymorphic deserialization without an allow-list (JAVA004); ASP.NET Core with no authorization anywhere (CSHARP001), deserialization that lets the payload pick the type (CSHARP002), the developer error page outside Development (CSHARP003) |
73 of 172 rules are preview; vibeaudit rules lists them. Preview findings are reported but do not
fail a build, with one exception: the severe classes (command injection, code execution, unsafe
deserialization, open redirects and XSS) still gate while in preview, so a high-impact bug is never let through
because its rule is new. Findings of any rule in Go, Java/Kotlin, C#, Rust and PHP code are preview; there, SQL
injection, command injection and XSS still gate. STABILITY.md explains how a rule leaves preview.
Every rule, with its severity, why it matters and how to fix it, is in the
rule catalogue (docs/RULES.md); vibeaudit rules prints the same list.
Routes are found for Flask, FastAPI, Django REST framework, Express, Fastify, Hono, Next.js (app and pages
routers, server actions), SvelteKit, Supabase edge functions and Firebase functions. Python is analysed on
its syntax tree with taint tracking (request data, model output and agent-tool arguments are followed into
SQL, shell, eval, file paths, URLs, templates, redirects and HTML); JavaScript and TypeScript get the same
checks on a lexer that keeps exact offsets.
Why another scanner
Good tools exist, and vibeaudit runs them when they are installed instead of replacing them:
| Tool | Great at | Misses in AI-built apps |
|---|---|---|
| gitleaks, trufflehog | secrets in files and history | secrets exposed by design: NEXT_PUBLIC_OPENAI_API_KEY, a service_role key in a client component, dangerouslyAllowBrowser: true |
| npm audit, pip-audit, OSV-Scanner | known CVEs in real packages | packages that do not exist (hallucinated) or appeared last week (slopsquatting) |
| Bandit, Semgrep, CodeQL | classic injection patterns | whether a route has any auth, whether model output reaches exec/SQL/HTML, RLS policies, Firebase rules |
| garak, promptfoo | red-teaming a running model | static code: user input in the system prompt, agent tools with shell access, missing max_tokens |
Install
vibeaudit is on PyPI. Python 3.9 or newer:
pip install vibeaudit
or, to keep it in its own environment:
pipx install vibeaudit
To install from this repository instead: pip install "vibeaudit @ git+https://github.com/Ashin-06/vibeaudit.git@v0.11.1".
Optional: if gitleaks, bandit,
osv-scanner or semgrep are on
your PATH, their findings are merged into the report and de-duplicated.
Commands
| Command | What it does |
|---|---|
vibeaudit scan [PATH] |
Audit a project (default: the current directory). |
vibeaudit watch [PATH] |
Scan, then re-scan after every change and print the findings that appeared (+) or were fixed (-); see Watch mode. |
vibeaudit review [PATH] --diff REV |
Pull request review comments, with suggested fixes, for the findings on changed lines; --post sends them to GitHub, --max-comments N caps them (see Pull request review). |
vibeaudit fix [PATH] |
Print a diff that fixes the findings with one obvious repair; --apply writes it (see Automatic fixes). |
vibeaudit studio [PATH] |
A local web UI: routes with auth and write badges, data flows, and the score by domain, category and rule (see Studio). |
vibeaudit test-exploit FINDING |
Build a non-destructive proof-of-concept request for one finding; a dry run by default, and --send sends it once to localhost only, never following redirects. |
vibeaudit explain FINDING |
Explain one finding (FILE:LINE, FILE:LINE:RULE or a fingerprint prefix): the rule, the code and the data flow that led to it (--format json, --path DIR for another project). |
vibeaudit routes [PATH] |
List every HTTP entry point with method, path, location and whether it has a visible auth check, writes data or calls a paid AI API (--format text|json|md). |
vibeaudit rules |
List every rule (--format text|json|md; JSON includes the why and fix text). |
vibeaudit feedback |
Record that a finding is a false positive or confirmed, or list recorded feedback (--path DIR for another project; see False positives). |
vibeaudit update |
Refresh the popular-package lists and the retired-model list (and the signed threat feed, once it is active; see Caching and updates); --self upgrades vibeaudit with pip; --export FILE [--osv PyPI,npm] and --import FILE carry the data to a machine without network (see Offline and air-gapped use). |
vibeaudit lsp |
Run the language server for editors: diagnostics and fix code actions (see Editors). |
vibeaudit mcp |
Run the MCP server for coding agents (see MCP server). |
Every command has --help.
vibeaudit scan options
| Option | Meaning |
|---|---|
-r, --report FORMAT[:PATH] |
Also write a report: json, sarif, md, html, prompt (a fix brief for AI assistants), badge (SVG), junit (test report XML for GitLab, CircleCI, Jenkins), codeclimate (GitLab Code Quality) or compliance:FRAMEWORK (see Compliance mapping). Repeatable; - as the path writes to stdout, as in -r json:-. Default path vibeaudit-report.<ext> (vibeaudit-compliance-FRAMEWORK.html). |
--fail-on SEVERITY |
Exit 1 when a finding is at least critical, high (default), medium or low; none never fails. |
--min-severity SEVERITY |
Hide findings below this severity (default low). |
--monorepo |
Scan every package as a project of its own, against its own manifest, with a score per package (see Monorepos). |
--include-preview |
Count findings of preview rules toward --fail-on and the score; they are always reported (see STABILITY.md). |
--no-gate-severe-preview |
Leave out the severe preview rules (command injection, eval, unsafe deserialization, open redirect, XSS), which by default count toward --fail-on and the score. |
--offline |
No network: skip registry and new-package lookups, and check known vulnerabilities against an imported OSV index if there is one (the typosquat check still runs). |
--exclude GLOB |
Skip matching paths (repeatable). A .vibeauditignore file with one glob per line works too. |
--disable RULE_OR_CATEGORY |
Switch off a rule (QA002) or a category (quality); repeatable, comma lists allowed. |
--baseline FILE |
Hide findings already present in an earlier JSON report. |
--write-baseline FILE |
Save all current findings as a baseline and exit 0. |
--fail-on-regression N |
Exit 1 if the whole project's score is more than N points below the score in the --baseline report. The score is computed before --baseline, --diff and --staged hide findings. |
--external auto|none|LIST |
Third-party scanners to run: those installed (default), none, or a list such as gitleaks,bandit. |
--staged |
Report only findings on lines staged for commit (pre-commit hooks). |
--diff REV |
Report only findings on lines changed since REV, for example origin/main (pull requests). |
--config FILE |
Read settings from this file instead of .vibeaudit.toml / pyproject.toml. |
--no-config |
Ignore the scanned project's settings and feedback files (use this for code you do not trust). |
--plugin FILE_OR_MODULE |
Load custom rules (repeatable; this runs that Python code). |
--llm gemini|openai|openrouter|anthropic |
Ask an LLM to triage findings (see LLM-assisted checks). |
--llm-model NAME, --llm-review, --llm-max-requests N |
Model name; also review the riskiest files for missed issues; hard cap on requests (default 10). |
--max-file-size BYTES |
Skip larger files (default 1,000,000). |
--file-timeout SECONDS |
Time each analysis step may spend on one file before it is stopped (default 30). |
--no-cache |
Do not read or write cached registry, vulnerability and LLM answers for this scan. |
-q, --quiet / --no-color |
Print only the summary / plain text. |
Exit codes: 0 nothing at the --fail-on level, 1 findings at that level, 2 usage error.
What is scanned: in a git checkout, tracked and untracked files that are not git-ignored (a local, ignored
.env is where secrets belong). Dependency and build folders (node_modules, .venv, dist, build,
.next, ...) are skipped, except that committed bundles under dist/, build/ and out/ and minified
files are still checked for secrets. In test files only real credentials are reported. Pinned versions
for the vulnerability check come from requirements*.txt and from lockfiles next to each manifest
(package-lock.json, pnpm-lock.yaml, yarn.lock, bun.lock, poetry.lock, uv.lock, pdm.lock,
Pipfile.lock).
A repository is untrusted input, so the scan is built to finish whatever it contains. A file that triggers
a bug is listed under meta.internal_errors in the JSON report while every other file is still scanned;
a file that exceeds --file-timeout or is nested too deeply to parse is listed under meta.skipped.
Symbolic links out of the project are not followed. Registry lookups retry timeouts and rate limits, stop
calling a host that is down, and never report a package as missing unless the registry said 404 or 410.
LLM-assisted checks (optional)
GEMINI_API_KEY=... vibeaudit scan . --llm gemini --llm-review --llm-max-requests 10
- Triage adds a verdict and a reason to medium and higher findings. The model can never remove or downgrade a finding, because the audited code is untrusted and may contain text aimed at the model. Secrets are redacted before anything is sent.
- Review adds issues the rules did not cover as
AI001, labelled model-generated and unverified. - Providers:
gemini(GEMINI_API_KEYorGOOGLE_API_KEY),openai(OPENAI_API_KEY;OPENAI_BASE_URLfor any compatible server such as Ollama),openrouter(OPENROUTER_API_KEY, defaultqwen/qwen4-max) andanthropic(ANTHROPIC_API_KEY). Verdicts are cached, so an unchanged finding costs no request the next time. - Model:
--llm-model(orllm_modelin the configuration) picks the model; otherwise theVIBEAUDIT_LLM_MODELenvironment variable does, and then the provider's default.
Monorepos
vibeaudit scan . --monorepo # or `monorepo = true` in .vibeaudit.toml
By default the whole repository is one project, so a rate limiter, CSRF middleware or a dependency declared in
one package counts for every package. With --monorepo every folder with its own manifest (package.json,
pyproject.toml, requirements*.txt, setup.py, setup.cfg, Pipfile) is scanned on its own, against
its own manifest and with only its own code as evidence; files outside every package (root scripts, CI,
infrastructure) are one more unit, and folders under tests or examples stay test code. Three rules are about
the repository, not a package (CI configuration QA002, the example env file DATA007 and a workspace lockfile
DEP005), so they come from one scan of the whole repository; the mode costs about two normal scans. Findings
keep their repository paths and fingerprints (baselines and feedback carry over), and the report adds a score
per package.
Studio
vibeaudit studio . # opens the browser; --no-browser prints the link, --port picks the port
studio scans the project and serves a web UI on this computer only: every route with its auth, write and
AI badges, the data flows from user input and model output to model prompts, databases, shells, HTML and
outgoing requests (traced as in explain), the score broken down by domain, category and rule, and the
code and explanation of every finding. "Re-scan" re-runs the scan, reusing what did not change.
The server binds 127.0.0.1 only and uses no external assets. Data requests need the random token from the printed link (it is in the URL fragment, so it never reaches a log or a Referer header), the Host header must be 127.0.0.1 or localhost (against DNS rebinding), the code view serves only files the scan listed, and a strict Content-Security-Policy lets only the page's own script run; repository text is never rendered as HTML. It reads the project and never writes to it.
Explaining a finding
vibeaudit explain app.py:19 # or app.py:19:LLM002, or a fingerprint prefix from a JSON report
explain shows the rule (what it means, why it matters, how to fix it), the code around the line and, when
the line uses request data or model output, the data flow traced back from it: each assignment the value
went through, up to where it came from (a request field, a route parameter, a model call, or in Python the
parameter of an ordinary function, whose callers are not followed). Data-flow traces cover Python and
JavaScript/TypeScript; for Go, Java/Kotlin, C#, Rust and PHP, explain shows the rule and the code only. The trace follows
variables within the function (Python) or the file (JavaScript and TypeScript), not calls into other
functions, and shows only the steps that lead to a source.
Data flow (traced back from the line, best effort):
from app.py:15 plan = client.chat.completions.create(...) (`plan` comes from model output)
via app.py:18 script = plan.choices[0].message.content (`script`)
to app.py:19 exec(script)
Watch mode
vibeaudit watch . # Ctrl+C to stop
vibeaudit watch . --interval 2 --min-severity medium
watch scans once, then checks for changed files every --interval seconds (default 1) and re-scans. It
keeps every unchanged file's text, parse trees, lexer state and per-file analysis between scans, and replays
the findings (and the routes) of per-file checks when nothing they depend on changed: the files they read,
the facts about other files they use (the names a module exports, whether an excluded path exists on disk)
and the project-wide facts they rely on (which functions call a model, which files sit behind auth
middleware, the configured helpers). Everything else runs again, so the result is always the same as a
fresh scan. A changed manifest or config file, or an added or removed file, replays nothing. On the
benchmark repositories a re-scan after one edit takes 0.13 to 0.46 s (median per repository; under 0.3 s on
four of five), against 0.5 to 4 s for a cold scan (tools/watch_latency.py). It runs offline, without
external tools, and honours the configuration, --exclude and --config like scan.
Automatic fixes (vibeaudit fix)
vibeaudit fix . # print a unified diff, change nothing
vibeaudit fix . --apply # write the changes
vibeaudit fix . --rule REL001 # only some rules (repeatable)
vibeaudit fix . > fixes.patch && git apply fixes.patch
fix scans the project the same way scan does (configuration, suppressions, baseline and feedback apply)
and rewrites the findings that have one obvious, deterministic repair:
| Rule | Codemod |
|---|---|
| REL001 | Adds timeout=10 to requests and urlopen calls. |
| LLM003 | Adds an output token limit of 4096: max_completion_tokens (Chat Completions), max_output_tokens (Responses, Gemini config, ChatGoogleGenerativeAI), max_tokens (LangChain and LiteLLM), maxOutputTokens or maxTokens for the AI SDK by its major version in package.json. |
| LLM011 | Wraps the assignment or return that parses model output in try/except ValueError (Python) or try/catch (JavaScript); the fallback value is None / null, which the caller must handle. |
| BE016, AUTH011 | Removes 'none' from an algorithms list, turns ignoreExpiration: true off, adds algorithms=[...] to PyJWT jwt.decode() (the algorithm the file signs with, else HS256), and narrows a mixed HMAC and RSA list to the algorithm the file signs with. |
| DATA004 | Float money columns become Numeric(12, 2) (SQLAlchemy), DecimalField(max_digits=12, decimal_places=2) (Django), Decimal (Prisma) or numeric(12, 2) (SQL). Create a migration afterwards; SQL files in migration folders are never edited. |
| BE012 | Replaces the wildcard or reflected CORS origin with an allow-list read from the CORS_ORIGINS environment variable (comma-separated), keeping credentials. |
| DATA013 | Gives a Redis key an expiry of one hour: ex=3600 (redis-py), { EX: 3600 } (node-redis), "EX", 3600 (ioredis) or { ex: 3600 } (Upstash) on set, added to an existing options object; for hset, lpush, sadd and the like, an expire(key, 3600) on the next line when the key is a plain name or string. Review the expiry in the diff. |
A file's change is kept only when the file still parses (ast.parse for Python, balanced brackets for
JavaScript and TypeScript). Files keep their line endings and byte order mark; files that are not UTF-8 or
mix line endings are left alone, and so is a file that changed on disk after it was read. Running fix
twice changes nothing the second time. Every finding a codemod could not rewrite is listed with the reason.
Patches from a model (optional). For findings without a codemod (auth checks, transactions, unverified
tokens), --llm asks the provider (see LLM-assisted checks) for the smallest
fix and writes it to --patch (default vibeaudit-fix.patch). It is never applied: a patch that does not
parse, reaches outside the lines the model was shown or contains redacted text is dropped, and the rest are
marked model-generated and unverified. --llm-model and --llm-max-requests work as for scan.
ANTHROPIC_API_KEY=... vibeaudit fix . --llm anthropic --patch auth-fixes.patch
Configuration
Settings live in .vibeaudit.toml or in a [tool.vibeaudit] table of pyproject.toml. Command-line flags
win over the file. Unknown keys and wrong types are errors, so a typo cannot quietly weaken a scan.
# .vibeaudit.toml
fail-on = "high"
min-severity = "low"
exclude = ["legacy/**", "scripts/one_off_*.py"]
disable = ["QA002"]
baseline = ".vibeaudit-baseline.json" # relative to this file
offline = false
external = "auto"
# Teach vibeaudit your project's own conventions
auth-helpers = ["require_org_member", "withWorkspaceAuth"] # count as an auth check in a route
sanitizers = ["clean_sql_input"] # their output is not tainted
public-paths = ["/api/public/*", "/api/survey-*"] # intentionally unauthenticated routes
Other keys: include-preview, gate-severe-preview, monorepo, llm, llm-model, llm-max-requests, max-file-size, file-timeout. The same settings apply
to vibeaudit scan, vibeaudit routes, vibeaudit feedback, the MCP server and the Python API, except llm,
which only vibeaudit scan reads: the LSP server, watch, studio, the MCP server and the API never send code to a
model provider because a repository's config asks for it. The file is part of the scanned repository and can
switch rules off: when you audit code you do not trust, use --no-config.
CI and pre-commit
--diff and --staged still analyse the whole project (auth middleware and helpers in unchanged files
count) but report only findings on added or changed lines.
GitHub Action, with results in the code scanning tab:
name: vibeaudit
on: [push, pull_request]
permissions:
contents: read
security-events: write
jobs:
audit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0 # needed for --diff
- uses: Ashin-06/vibeaudit@v0.11.1
with:
fail-on: high
args: --diff origin/main # optional: only changed lines
- uses: github/codeql-action/upload-sarif@v3
if: always()
with:
sarif_file: vibeaudit.sarif
Inputs: path (default .), fail-on (default high), sarif (default vibeaudit.sarif), args (extra
flags, split on spaces), setup-python (true/false), python-version, review (true posts review
comments on pull requests) and github-token (default: the workflow token).
Pull request review
With review: "true" the action also comments on the pull request: one review comment per finding on a
changed line (the 30 most severe), with why it matters, and a one-click GitHub suggestion when a
vibeaudit fix codemod rewrites it inside the diff. A re-run does not repeat comments it already posted.
The job needs pull-requests: write; on pull requests from forks the token is read-only and the review is
skipped with a warning. Use pull_request, not pull_request_target, so the pull request's code never runs
with a write token.
on: pull_request
permissions:
contents: read
pull-requests: write
jobs:
review:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0
- uses: Ashin-06/vibeaudit@v0.11.1
with:
review: "true"
args: --diff origin/${{ github.base_ref }} --baseline .vibeaudit-baseline.json --fail-on-regression 5
vibeaudit review . --diff origin/main prints the same review as JSON without posting it. To keep the
overall score from slipping, commit a baseline (vibeaudit scan . --write-baseline .vibeaudit-baseline.json)
and pass --fail-on-regression N: the build fails when the whole project's score drops more than N points
below the baseline's, even if every new finding is below --fail-on.
GitLab CI: include the template, pinned to a release. Merge request pipelines report only changed lines;
findings show in the merge request's Code Quality widget and in the pipeline's Tests tab. Set
VIBEAUDIT_FAIL_ON or VIBEAUDIT_ARGS in your own variables: to change it.
include:
- remote: https://raw.githubusercontent.com/Ashin-06/vibeaudit/v0.11.1/ci/gitlab-ci.yml
CircleCI: copy ci/circleci.yml into .circleci/config.yml (or merge its job into
yours). Branches other than VIBEAUDIT_BASE (default main) report only changed lines; findings show in the
Tests tab, and the reports are stored even when findings fail the job.
Any other CI: run vibeaudit scan . --fail-on high -r sarif:results.sarif (or -r junit:vibeaudit.xml
for a test report) and keep the exit code.
pre-commit:
# .pre-commit-config.yaml
repos:
- repo: https://github.com/Ashin-06/vibeaudit
rev: v0.11.1
hooks:
- id: vibeaudit
The vibeaudit hook analyses the whole project and reports findings on the staged lines, which takes a
few seconds on a medium project. For a hook that never slows a commit down, use vibeaudit-secrets
instead (or as well, with vibeaudit in CI): it runs only the secret checks, which need nothing but the
file itself, on the staged content of the changed files, with one git diff call and without scanning the
rest of the project. On a 300-file repository it takes about 0.2 s on Windows, most of it Python and git
starting, against 1 to 5.5 s for the full hook on medium corpus repositories (tools/hook_latency.py). The
same command is vibeaudit-hook. Settings, vibeaudit: ignore comments, the baseline and feedback apply.
Editors (VS Code, Cursor and other LSP clients)
vibeaudit lsp is a Language Server Protocol server
on stdio, written with the standard library only. It scans the workspace like vibeaudit scan (offline,
no external tools, with the workspace's configuration) and publishes every finding as a diagnostic, also for
unsaved changes. It keeps unchanged files between scans as watch does, and re-scans 0.25 seconds after the
last edit. The vibeaudit fix codemods are offered as quick fixes, plus a "fix all" action per file. The
server never writes a file: the editor applies the edits you accept. It accepts only file: URIs inside
the workspace folder and caps message sizes.
editors/vscode is a thin VS Code / Cursor extension that starts the server. CI builds it as a .vsix
artifact (it is not published); install it with Extensions: Install from VSIX.... Its
vibeaudit.serverCommand and vibeaudit.serverArgs settings are machine-scoped and it does not run in
untrusted workspaces, so a cloned repository cannot make it start another program. Other editors can run
vibeaudit lsp directly, for example in Neovim:
vim.lsp.start({ name = "vibeaudit", cmd = { "vibeaudit", "lsp" }, root_dir = vim.fn.getcwd() })
MCP server for coding agents
vibeaudit mcp runs a Model Context Protocol server on stdio, so Claude
Code, Cursor and other agents can audit their own changes before handing them back. All tools are
read-only:
| Tool | What it does |
|---|---|
scan_project |
Audit a directory (optionally only changed lines: changes: "staged" or a git revision). |
scan_file |
Audit one file in the context of its project. |
audit_current_file |
Audit the file being edited, also from unsaved or not yet written content; repeated calls reuse unchanged files. |
list_routes |
The routes inventory. |
check_package |
Before adding a dependency: does it exist, is it brand new, is it a typosquat, known vulnerabilities. |
verify_package |
check_package plus a verdict (ok, caution, block) with reasons, the real package for a Python import name (cv2 is opencv-python) and whether the project already declares it. |
simulate_migration |
Dry-run a PostgreSQL migration (path to a .sql file, or sql): per statement the lock, what it blocks, table rewrites, data loss and a safer alternative. |
explain_rule |
What a rule means and how to fix it. |
auto_repair |
The vibeaudit fix diff for a directory (optionally one file or some rules); it writes nothing. |
{ "mcpServers": { "vibeaudit": { "command": "vibeaudit", "args": ["mcp"] } } }
claude mcp add vibeaudit -- vibeaudit mcp
Custom rules and the Python API
# org_rules.py
from vibeaudit import route_rule, file_rule
@route_rule("ORG001", "Billing route without audit log", severity="high", fix="Call audit_log().")
def billing_audit(route):
if route.path.startswith("/api/billing") and "audit_log(" not in route.body:
return f"{route.path} changes billing without calling audit_log()."
@file_rule("ORG002", "print() in production code", severity="low", kinds=["py"])
def no_print(source_file):
for number, line in enumerate(source_file.lines, 1):
if line.strip().startswith("print("):
yield number, "Use the logger instead of print()."
vibeaudit scan . --plugin org_rules.py
rule(...) registers a project-wide check that receives the project and the route table. Custom findings
appear in every report format and can be suppressed and baselined like built-in ones. Plugins run Python
code, so they are loaded only from --plugin or your own imports, never from the scanned repository.
from vibeaudit import scan, project_options
result = scan("path/to/project", project_options("path/to/project", offline=True))
print(result.score, result.grade, result.domain_scores)
for f in result.findings:
print(f.severity, f.rule_id, f.file, f.line, f.message)
for r in result.routes:
print(r.methods, r.path, r.has_auth)
scan(path) without options, and project_options(path, **overrides), apply the project's configuration;
an explicit Options(...) is used exactly as given. scan() may be called from several threads; scans
then run one at a time.
Compliance mapping
Every rule is mapped to the controls its findings are evidence for, in
src/vibeaudit/data/compliance_map.json: the OWASP Top 10 2021, the
OWASP Top 10 for LLM Applications 2025, themes of the EU AI Act (Art. 9, 10, 14 and 15), the SOC 2 Trust Services
Criteria and, where they apply, the HIPAA Security and Privacy Rules and the GDPR (Art. 5, 25 and 32). Rules
that no control fits (accessibility and other user-experience checks, for example) say so in a note.
vibeaudit scan . -r compliance:soc2 # writes vibeaudit-compliance-soc2.html
vibeaudit scan . -r compliance:gdpr:reports/gdpr.html
Frameworks: owasp, owasp-llm, eu-ai-act, soc2, hipaa, gdpr. The report is one self-contained HTML
file (no scripts, fonts or other external assets) with a print stylesheet, so it prints or saves to PDF as
is. For every control it shows a status (findings, no findings, not checked because the mapped rules were
disabled, or not covered by any rule) and links each piece of evidence to its finding.
This is an aid for reviewers, not a certification, an audit result or legal advice. "No findings" means the automated checks found nothing, not that the control is met, and most controls need evidence that code scanning cannot provide. The SOC 2, HIPAA, GDPR and EU AI Act reports list only the controls code scanning can speak to.
Scoring
Each finding has a severity. The overall score is 100 minus 25 per critical, 10 per high, 3 per medium and 1 per low finding, floored at 0. The grade is F if there is any critical finding, otherwise A (90 and up), B (75), C (60), D (40) or F.
The same formula gives a score per domain, so you can see where the problems are:
| Domain | Categories |
|---|---|
| Security | secrets, supply chain, backend, frontend, auth, config |
| AI | LLM and agents, LLM review |
| Quality | code completeness, frontend-backend contracts, custom rules |
| Data, Reliability, UI | one category each |
The terminal, JSON (summary.domains), Markdown and HTML reports all show them. A score is a summary for
tracking progress, not a security guarantee: read the findings.
Findings of preview rules (marked (preview), "preview": true in JSON) are reported but left out of
the score, the grade and --fail-on, unless you pass --include-preview. The exception is the severe
preview rules (command injection, eval, unsafe deserialization, open redirect and XSS, marked
"gates_in_preview": true): a miss there costs more than a false alarm, so they count by default;
--no-gate-severe-preview leaves them out. A rule is in preview while its
measured precision is below 90% or it is new; see STABILITY.md.
False positives and suppressing findings
Static analysis guesses, and some findings will be wrong for your project. In order of preference:
-
Teach it your conventions in the configuration file:
auth-helpersfor your own auth checks,sanitizersfor your own cleaning functions,public-pathsfor routes that are meant to be open. -
Record a verdict, shared with your team through git:
vibeaudit feedback false-positive AUTH001 app/api/internal.py:12 --note "only reachable from the VPN"
vibeaudit feedback list
Verdicts live in
.vibeaudit/feedback.json. Later scans hide findings marked as false positives (the report says how many); entries follow the code when lines move. Absolute paths and paths copied from an editor work;--configand--pluginwork as forscan. -
Inline:
# vibeaudit: ignore[AUTH001](or// vibeaudit: ignore[AUTH001], or without the bracket for every rule) on the line or the line above. -
Paths, rules, existing debt:
--excludeor.vibeauditignore,--disable, and--baselineto see only new findings.
An empty except/catch with an explanatory comment is treated as deliberate. If a rule is wrong in a way
that is not specific to your code, please open an issue with a minimal example.
Offline and air-gapped use
--offline makes no network connection (a test runs every offline command with the network disabled). Only
--llm, which you ask for explicitly, still sends code to its model provider. The popular-package lists used
by the typosquat check ship with vibeaudit. To use fresh data and check known vulnerabilities without network,
make a bundle on a connected machine and import it:
vibeaudit update --export vibeaudit-data.zip --osv PyPI,npm # connected machine (downloads OSV's dumps)
vibeaudit update --import vibeaudit-data.zip # air-gapped machine
vibeaudit scan . --offline
The bundle has the data files and, with --osv, a vulnerability index per ecosystem built from
OSV's public dumps: for every package, each advisory's listed affected versions and fixed
ranges. An offline scan reports a pinned version that an advisory lists (DEP004, with the version to upgrade
to) and says in its notes which index date it used, or that vulnerabilities were not checked. Advisories that
give only version ranges and no version list are left out of the index (the export says how many). --import
checks every file against the SHA-256 in the bundle's manifest, accepts only vibeaudit's own data files and
validates their contents before it writes anything.
Caching and updates
Registry answers and vulnerability advisories are cached for a day ("package does not exist" for an hour,
because a name can be registered at any time); failed lookups are never cached. The cache is in
%LOCALAPPDATA%\vibeaudit, ~/Library/Caches/vibeaudit or ~/.cache/vibeaudit (VIBEAUDIT_CACHE_DIR
overrides it). --no-cache or VIBEAUDIT_NO_CACHE=1 turns it off.
vibeaudit update downloads the latest popular-package lists (used for typosquat checks) and retired-model
list (LLM012) from this repository; a file is used only if it passes validation. Set GITHUB_TOKEN if you
hit GitHub's rate limit.
The signed feed is not active yet. No release ships the feed's public key and nothing is published on the
threat-feed branch until the maintainer does the one-time key setup in docs/THREAT_FEED.md, so for now
vibeaudit update installs no feed and scans use only the data packaged with vibeaudit.
Once it is set up, vibeaudit update also downloads the signed threat feed: slopsquatted package names (DEP001, also offline), a model sunset
calendar with retirement dates and replacements (LLM012) and prompt-injection patterns for agent instruction
files such as CLAUDE.md and .cursorrules (LLM025). The same data ships with vibeaudit, so every check works
offline; the feed only adds what was published since the release. A feed is used only when its signature
verifies against the key shipped with vibeaudit and it has not expired (30 days); otherwise vibeaudit warns and
uses the shipped data. Bundles made with --export carry the feed too. How to submit an entry, and how the
feed is signed: docs/THREAT_FEED.md.
Limits and what is not covered
How it reads code:
- Data flow is followed inside a function and, in Python, into the functions it calls (same file,
selfmethods and functions imported from the project, up to 3 calls deep); methods of other objects and dynamic calls are not followed. In JavaScript and TypeScript, calls are followed into functions of the same file and named imports from project files (3 calls deep); methods of objects and default imports are not. - JavaScript and TypeScript are read with a lexer and patterns, not a full parser.
- Java and Kotlin: Spring and JAX-RS/Quarkus routes feed the route checks, with @PreAuthorize, @Secured, @RolesAllowed and the Spring Security filter chain counted as authentication (JAVA001 when there is no filter chain). Mappings inherited from annotated interfaces are not followed yet.
- "Authenticated" means an auth check is visible in the route or its helpers, not that the check is correct.
- Rate limiting and input validation are detected project-wide, so one limiter anywhere counts.
- Router prefixes are resolved for FastAPI (
APIRouter(prefix=...),include_router), Flask blueprints (url_prefix,register_blueprint) and Express (app.use("/api", router)), through imports and nested routers, when each prefix is a string literal and each router is mounted once. The frontend-backend contract rules run when every route's full path is known (or the project has no prefixes). Django URLconfs, NestJS and Remix routes are only partly covered. - UI rules cover React, Next.js, Vue and Svelte single-file components (templates and
onMounted/onMounthooks) and HTML templates; Astro files get only the alt-text and placeholder checks. - Go (net/http, Gin, Echo, Fiber, Chi, GORM), Java/Kotlin (Spring Boot, Quarkus), C#/.NET (ASP.NET Core, EF Core), Rust (Actix, Axum) and PHP (Laravel, Symfony) get route/auth discovery, SQL and command injection (BE001, BE002), template XSS (FE001), loop N+1 queries (DATA001) and HTTP client timeout checks (REL001). Go, Java/Kotlin, C#, Rust and PHP also follow request data into file paths, outbound URLs and redirects and have a labelled corpus; findings in these languages stay preview until measured on held-out repositories (injection and XSS still gate).
vibeaudit test-exploit FINDING --url http://127.0.0.1:8000 [--send]builds (and with--send, runs) a single non-destructive localhost proof-of-concept request without following redirects;vibeaudit.runtimeoffers double-gated staging-only observability middleware for FastAPI and Express (VIBEAUDIT_RUNTIME=1).- The per-file time budget is checked between regex matches and statements; a single regex step cannot be interrupted, which is why every pattern is tested for linear time.
Not covered (see ROADMAP.md; contributions welcome):
- Languages not yet wired beyond the list above: Ruby.
- Infrastructure as code beyond the checks listed above (AWS only for Terraform and CloudFormation; no Azure or GCP resources, no Terraform modules or variables resolved, Helm templates read as plain YAML).
- XML external entities (XXE) and other XML parser settings.
- Prompt injection and jailbreaks against a live model (use promptfoo or garak for live model red-teaming).
Accuracy
Every rule is measured on a labelled benchmark: 106 public repositories pinned to exact commits, where each of the 3,687 findings was read in the code and labelled true or false, plus seeded examples for recall. 80 repositories were used to find and fix false positives; 26 are held out and never used for tuning.
| Precision | Tuned split | Held-out split |
|---|---|---|
Stable findings (what --fail-on acts on) |
94.4% (1339/1418) | 94.1% (270/287) |
| All rules, preview included | 79.6% (2429/3052) | 71.0% (451/635) |
Preview rules are reported but do not gate, and most false positives are theirs. Recall, F1 and the misses: docs/BENCHMARK.md.
Precision per rule on both splits, seeded recall and the status of every rule are in benchmark/RESULTS.md. Every stable rule with at least 5 labelled findings is at 90% or above; rules below that are in preview, and a rule leaves preview only on its held-out precision. CI fails when a rule's precision drops, a labelled true positive disappears or seeded recall drops.
Reproduce it (the repositories are cloned into ~/.cache/vibeaudit-bench, never into this one):
python tools/benchmark.py run --json results.json
How the corpus was chosen and how findings are labelled: benchmark/README.md.
Precision, recall, F1, the misses and the comparison with Semgrep and Bandit:
docs/BENCHMARK.md (python tools/benchmark.py report).
Help wanted: independent labellers. All labels so far are the maintainers' own. If you can spend an hour or two labelling a blind sample of 150 findings, see docs/LABELLING.md; every set of answers is published with its agreement score (Cohen's kappa), disagreements included.
A sample report for a deliberately vulnerable app (HTML and the six compliance reports) is published from
.github/workflows/pages.yml.
How it was validated
- Sample apps. Five deliberately vulnerable apps (Flask + OpenAI, Next.js + Supabase, Express + MongoDB, FastAPI + LangChain agent with an MCP config, config files only) and one clean app. Every expected finding is marked in the code; the tests compare the exact set of (file, line, rule), so a missed finding or a false alarm fails, and the clean app must produce nothing. Fake credentials are generated at test time.
- Property tests. Each vulnerable app is changed in ways that keep it vulnerable (lines shifted, variables renamed, CRLF, moved into a monorepo, routes moved into a blueprint) and every finding must stay; 33 safe rewrites (parameterise the query, add the auth check, ...) must each remove exactly the targeted finding.
- Fuzzing and hostile input. Random and malformed files with fixed seeds (garbage code, binary files, odd encodings, 3,000-level nesting, 1 MB single lines, broken manifests and lockfiles, symlink loops, unreadable files): the scan must not crash, hang or write invalid JSON or SARIF. Every regular expression is timed against input designed to make it backtrack.
- Scale. A generated 10,000-file repository scans in about a minute (59 s on a Windows laptop, 130 MB peak traced memory); CI runs 2,000 files on every platform and 10,000 on Linux.
- Real projects. The rules were run on 106 public repositories (benchmark/corpus.json), from AI chat apps and Next.js SaaS templates to Flask and Werkzeug themselves and deliberately vulnerable apps, and every finding was read. False alarms were fixed at the root with a regression test; confirmed problems included an unauthenticated route in a popular chat UI that loads any user's API key by id with the service-role key.
- Four external audits. Each finding was reproduced with a failing test before it was fixed; see the CHANGELOG.
- Packaging. The built wheel was installed in fresh environments on Python 3.9, 3.11, 3.12 and 3.13 and every command was run; CI runs the tests on 3.10 to 3.13 on Linux, macOS and Windows and on 3.9 on macOS and Windows (the current Ubuntu runner has no 3.9), and also from a plain checkout without installing.
Verifying a release
Each GitHub release has the wheel, the sdist and a
CycloneDX SBOM (vibeaudit-VERSION.cdx.json), built by the release workflow from the tagged commit. Their
build provenance is signed with GitHub artifact attestations (Sigstore). With the GitHub CLI:
gh attestation verify vibeaudit-0.11.1-py3-none-any.whl --repo Ashin-06/vibeaudit
Run the same for the .tar.gz and the .cdx.json file. The check passes only if the file was built by
this repository's release workflow; --format json shows the commit and workflow it came from. The SBOM
lists what the wheel installs (vibeaudit has no runtime dependencies on Python 3.11 and later, and
tomli before that). PyPI receives these same attested files through trusted publishing; to install a wheel you verified
yourself, run pip install vibeaudit-0.11.1-py3-none-any.whl.
Contributing, security and license
- Contributing: see CONTRIBUTING.md. Bug reports with a minimal example are the most useful contribution, especially false positives and missed findings.
- Security issues in vibeaudit itself: see SECURITY.md; please do not open a public issue.
- License: MIT, see LICENSE. Popular-package lists derive from top-pypi-packages and npm-high-impact; vulnerability data comes from OSV.dev.
Metadata
Release files for vibeaudit 0.11.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| vibeaudit-0.11.1.tar.gz | 754.6 kB | With provenance | Details for vibeaudit-0.11.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| vibeaudit-0.11.1-py3-none-any.whl | Python 3 | none | any | Details for vibeaudit-0.11.1-py3-none-any.whl |
Total release size: 1.3 MB
Release files / vibeaudit-0.11.1.tar.gz
| Download URL | vibeaudit-0.11.1.tar.gz |
|---|---|
| Size | 754.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6993609e7183d02218bcef276c533bd497993a3f822e5bbbbc19f71f497f53a8
|
|
BLAKE2b-256 checksum How to use checksums |
897586e624a4139d58060e45df2770bac5530c93ae1454bc6991b6b3e3266f1d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.
Transparency logRelease files / vibeaudit-0.11.1-py3-none-any.whl
| Download URL | vibeaudit-0.11.1-py3-none-any.whl |
|---|---|
| Size | 593.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b355b66e59273b9bc96480b29d78ba87c1ce9a23b0680c99471d6379f33af80e
|
|
BLAKE2b-256 checksum How to use checksums |
711209ec2ef2f1b203638dec54bf184f57d4af05b9867eb6cefa87740c9c3107
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.
Transparency logSecurity for vibeaudit 0.11.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for vibeaudit-0.11.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.