slopaudit

Check AI-written code before you trust it. slopaudit is an open-source command-line tool for Python that finds the problems AI assistants and LLM-powered apps tend to introduce, and that general scanners were not built to look for.

• Is it safe? Does model output reach eval, a shell or a SQL query? Does untrusted input end up in a system prompt? Does an agent tool hand the model a shell?
• Is it real? Does that imported package exist, or is it invented (or one typo from a popular one)?
• Is it finished? Stubs, "rest of code here" comments, swallowed errors, truncated files.
• Is it what you asked for? Does the code meet each requirement in your spec, and did the change delete or weaken tests to get there?

It runs on your machine, never executes the code it scans, needs no account or API key, and has no telemetry.

Install

You need Python 3.10 or newer. Pick one:

pipx install slopaudit             # recommended: an isolated command-line tool
uv tool install slopaudit          # same idea, if you use uv
python3 -m pip install slopaudit   # inside a virtual environment

Then check it works:

slopaudit --version

Just try it, without installing anything:

pipx run slopaudit scan .          # or: uvx slopaudit scan .

In CI, no install step is needed. Use the GitHub Action:

# .github/workflows/slopaudit.yml
name: slopaudit
on: [pull_request]
permissions: { contents: read, security-events: write }
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: patsypppe/slopaudit@v1.0.0

Before every commit, with pre-commit:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/patsypppe/slopaudit
    rev: v1.0.0
    hooks:
      - id: slopaudit

See it work

$ slopaudit scan examples/vulnerable_agent
vulnerable_agent/app.py
  16   HIGH   LLM-08  Tool `run_command` passes a model-chosen argument to subprocess.run() (shell)
        ↳ line 14: tool argument `command` is chosen by the model
        ↳ line 16: passed to subprocess.run()
  21   HIGH   LLM-01  Untrusted input reaches the system prompt of client.messages.create()
        ↳ line 20: untrusted input request.args
        ↳ line 20: stored in `topic`
        ↳ line 21: placed in the system prompt of client.messages.create()
  27   CRITICAL   LLM-02  Model output reaches eval() without validation
        ↳ line 21: model output from client.messages.create()
        ↳ line 21: stored in `reply`
        ↳ line 27: passed to eval()
  30   MEDIUM   INC-02  `parse_invoice` has no implementation (pass)
  34   CRITICAL   SEC-01  Hardcoded credential assignment (q9X$…(21 chars))

1 files scanned: 2 critical, 2 high, 1 medium

Every flow finding shows the path the data took, so you can judge it in seconds. Each finding also has a fix: line and a longer explanation: slopaudit explain LLM-02. No model is involved: slopaudit reads the code's structure and follows values from where they come from to where they are used (how it works).

Quickstart

slopaudit scan .                              # scan the current folder
slopaudit scan . --no-network                 # skip the PyPI package check
slopaudit scan --staged                       # only what you are about to commit
slopaudit scan . --format sarif -o out.sarif  # for GitHub code scanning
slopaudit scan src --spec SPEC.md             # check the code against a requirements file
slopaudit explain LLM-02                      # why a rule exists, and how to fix it

From Python, for example to check an assistant's answer before saving it:

import slopaudit

blockers = [f for f in slopaudit.scan_source(answer) if f.severity >= slopaudit.Severity.HIGH]

New to it? The getting-started guide takes five minutes, and has a ten-line test file that shows every kind of finding. Adopting on a big codebase? Start with a baseline so only new problems fail.

What it checks

The full list, with severity and fix advice for each rule, is the rules reference. Beyond scanning, there are optional model-assisted commands: gen-tests and fix (a dry run unless you pass --apply).

How well does it work?

Three checks, each reproducible from this repository. None is a guarantee; the caveats matter.

1. Labelled corpus (slopaudit bench, 34 files). Every rule scores 1.00 precision; recall is 0.91, with two labelled misses (a value that crosses a function boundary, or goes through self). The corpus is small and was used to tune the rules, so treat it as a regression gate.

2. Other scanners on that corpus (full report): files flagged at medium severity or higher, run as external programs.

The corpus targets what slopaudit was built for, so it flatters slopaudit. Bandit and Semgrep are broader tools not designed for these cases, and Semgrep's ai rules are SDK best-practice lints. Semgrep also catches the two cross-function cases slopaudit misses, because it flags any dynamic eval.

3. Mature packages as a false-positive control (report): 1.5 million lines across twelve well-reviewed packages, including the OpenAI, Anthropic, LangChain and LiteLLM SDKs. 0 critical or high findings after tuning. The first run had 122 critical findings and about 900 medium ones; fixing them is in the changelog. About 25 medium and a few hundred low findings remain. This measures noise only: those packages are not AI-generated.

Honest limits

• A clean scan is not proof of safety. The flow rules follow values inside one function. They miss a value returned from a helper and eval-ed elsewhere, and values stored on self (how it works).
• Without a model, the spec check compares names and words. It can show code exists, not that it behaves correctly. In the demo it reports a requirement as met while the code does the opposite.
• Experimental parts. scan --spec --llm, gen-tests, fix and the Docker sandbox are tested against local stand-ins, not live services, so they are outside the 1.0 stability promise (what 1.0 promises).
• Python only. JavaScript and TypeScript files are skipped, and the scan says so when it finds nothing to read.
• Linux CI only. Installing and running on macOS has been checked by hand; Windows is untested (#6).

Documentation

Contributing

Contributions are welcome: rules, false-positive reports, docs and tests. The most valuable thing you can do is report a false positive or a miss with the smallest code that shows it; every confirmed report becomes a regression test.

Looking for something to work on? The contributor roadmap lists every open issue by area and difficulty. Each one says why it matters, which files to change, and when it is done:

Comment on an issue before you start so nobody duplicates work. Setup takes one click in Codespaces, or make install locally. Start with CONTRIBUTING.md and writing a rule, and please follow the code of conduct. Questions go in Discussions or issues; security problems in SECURITY.md.

License

MIT. slopaudit is independent work: it takes no code, rules, messages or structure from other tools. The clean-room record explains how, and why other scanners are only ever run, never copied. Thanks to the authors of Bandit, Semgrep and vibechck for the prior art that made the comparison possible.