depverify

Verifies Python/PyPI dependencies referenced in LLM-generated code against reality. Advisory only: it produces a JSON report. It never installs, deletes, halts, or quarantines anything.

Why

LLM coding models hallucinate package names. USENIX Security 2025 (Spracklen et al.) measured ~19.7% of recommended packages as hallucinated across 16 models (~5% commercial, ~21% open-source); a May 2026 replication found frontier models compressed to ~4.6-6.1%, but quantized local models remain far worse. Attackers register hallucinated names on PyPI with malware ("slopsquatting"). depverify checks whether a name an LLM told you to pip install or import actually exists, and, if it does, whether it looks reputable.

Install

pip install depverify              # core
pip install depverify[flask]       # + flask itself, for the HTTP integration below

depverify[flask] installs the flask dependency only. The HTTP integration itself (integrations/cortexfeed/flask_blueprint.py) is source-only, deliberately not part of the installed depverify package -- see Flask HTTP integration below.

Dev: pip install -e ".[flask,dev]" (pytest, pandas, flask).

Usage

depverify check path/to/answer.txt
cat answer.txt | depverify check -
depverify check answer.txt --json
depverify check answer.txt --symbols

A successful check's exit code is always 0, regardless of what it finds (NOT_FOUND, suspicious, etc.) -- this is an advisory tool, not a linter that fails your build over the contents of a report. That guarantee covers verdicts, not invocation: a CLI usage error (bad arguments, a missing/unreadable input file, non-UTF-8 input) is reported as a clean one-line message on stderr and exits non-zero, the same way depverify check with a missing required argument already does -- it is not itself a verdict, so it is not covered by the "always 0" guarantee.

--symbols additionally checks statically-resolvable attribute chains (e.g. pandas.DataFrame.flatten) against packages already installed in the local environment. Security note: this is the one depverify feature that imports locally installed code, which can execute that code's import-time side effects -- see Security note: --symbols and local code execution before enabling it. Off by default; normal depverify check never imports or executes anything locally installed.

Library

from depverify import verify_text

report = verify_text(llm_answer_text)
print(report.to_json())

Verdict model

Per detected dependency:

verdict ∈ EXISTS | NOT_FOUND | STDLIB_SKIPPED | LOCAL_OR_UNKNOWN | CANT_VERIFY

EXISTS packages additionally carry a risk object: level ∈ ok | suspicious, with reasons ⊆ {young_package, low_downloads, near_name:<popular-package>, version_missing}.

Existence and reputation are orthogonal. A package that EXISTS can still be suspicious. depverify never folds "sketchy" into "missing" -- those are different questions with different implications.

The core asymmetry

A name that 404s on PyPI is interpreted differently depending on how it was detected:

• Detected via an explicit pip install <name> line -> NOT_FOUND. The LLM told you to install this; it doesn't exist; that's the signal slopsquatting exploits.
• Detected via import <name> -> LOCAL_OR_UNKNOWN. It might be your own module, a relative import target, or a name that isn't on PyPI at all for a legitimate reason. Flagging every unresolvable import as "not found" would bury real warnings in false positives from ordinary project code (import myapp.models, etc.).

Report shape

{
  "packages": [
    {"name_raw": "cv2", "resolved": "opencv-python", "source": "import",
     "verdict": "EXISTS", "risk": {"level": "ok", "reasons": []}},
    {"name_raw": "pdfreader-pro", "resolved": "pdfreader-pro", "source": "pip_install",
     "verdict": "NOT_FOUND", "risk": null}
  ],
  "summary": {"checked": 2, "not_found": 1, "suspicious": 0, "cant_verify": 0}
}

summary.cant_verify counts only package-level CANT_VERIFY verdicts. It does not include symbol-level CANT_VERIFY results (e.g. a --symbols chain that couldn't be checked because the package isn't installed locally) -- those live under each package's own symbols list (see --symbols below) and are a separate count from the top-level summary. This is deliberate: summary() predates the symbol-checking feature and was kept unchanged so existing consumers of Report.summary()/to_dict() see byte-identical output when check_symbols is left at its default.

Reputation signals (EXISTS packages only)

• Age: days since the earliest PyPI release.
• Downloads: last-30-day count from pypistats.org. If that API is unreachable or rate-limited, depverify silently omits download-based reasons -- a missing download count is never treated as CANT_VERIFY.
• Near-name: difflib.SequenceMatcher.ratio() >= 0.88 (tunable, see reputation.py) against a vendored list of popular package names (depverify/top_packages.json), flagged as near_name:<popular-package> when the candidate itself isn't already a popular package.

Default suspicion rule (a tunable default, not ground truth): suspicious if (age < 60 days AND downloads < 1000/month) OR any near_name hit. Tune the constants in depverify/reputation.py for your risk tolerance.

What this does NOT do

• No package-content scanning. depverify checks whether a name exists and looks reputable by metadata -- it does not download, sandbox, or static-analyze package code. For that, see dedicated tools like Socket or Snyk.
• No blocking. Nothing here gates a pip install, a CI job, or an LLM response. It's a report.
• No npm/yarn. Python/PyPI only, by design.
• No LLM-based judging. Every verdict is deterministic: extraction is regex/AST, existence is a PyPI lookup, reputation is arithmetic over metadata. No model calls anywhere in this codebase.

Security note: --symbols and local code execution

Normal depverify check (no --symbols) never imports or executes any locally installed package code. Existence checks are PyPI metadata lookups over HTTP; reputation checks are pypistats.org lookups and arithmetic. Nothing in that path touches your local Python environment's installed packages, and nothing ever executes the LLM-generated code being scanned.

--symbols is different, and deliberately scoped narrowly because of it:

• To check whether an attribute chain like pandas.DataFrame.flatten really exists, --symbols uses importlib.import_module() to import the already-installed package and getattr() to walk the chain. Importing a Python module runs that module's top-level code -- this is ordinary Python behavior, not something depverify adds, but it means --symbols is the one code path in this tool that executes local code as a side effect of scanning.
• This is not the same as executing the LLM's code. --symbols never runs the snippet being checked; it only imports packages by name using importlib, and only ever calls getattr/hasattr on the resulting module or class objects -- never instantiates a class, never calls a function or method. (Confirmed in tests/test_symbols.py: walking a chain that reaches a property or method never triggers the property getter, the method body, or the class's __init__.)
• --symbols never installs anything. If a package exists on PyPI but is not already installed locally, the result is CANT_VERIFY, not an install-then-import. depverify's "never installs, deletes, halts, or quarantines anything" guarantee holds for --symbols too.
• The residual risk: if a malicious or already-compromised package happens to be installed in the same environment running depverify, --symbols will trigger that package's import-time code the same way a plain import thatpackage anywhere else in that environment would. depverify does not sandbox, isolate, or vet locally installed packages before importing them for a symbol check.
• Consequently: only enable --symbols in an environment where the installed package set is already trusted -- the same trust you'd already extend to running python -c "import <installed package>" in that environment. Do not run --symbols as a way to safely inspect an environment whose installed packages you don't already trust, and do not present or rely on --symbols as a sandbox or security boundary around untrusted local installs -- it is not one.

Limitations

• Dynamic imports are not detected. importlib.import_module("name") string literals are invisible to the AST/regex extraction in extract.py. Only literal import x / from x import y statements and pip install lines are found.
• Only pip-style dependency declarations are parsed. setup.py, pyproject.toml ([project.dependencies]), Conda environment files, and Poetry's pyproject.toml dependency tables are not parsed.
• The import-name -> distribution-name mapping table is incomplete by nature. depverify/mapping_table.py is hand-curated (71 entries as of this writing) and will always miss some real-world aliases. Unmapped import names fall back to querying PyPI with the import name as-is, which works for the common case (import name == distribution name) but not for every alias.
• Verification is point-in-time. A name that 404s right now can be registered on PyPI minutes later -- including by an attacker watching for exactly this kind of hallucinated name (slopsquatting). Nothing here caches a "safe" verdict indefinitely; the cache TTLs (24h for EXISTS, 6h for NOT_FOUND) reflect that names that don't exist yet are the more time-sensitive case.
• EXISTS ≠ safe: compromised legitimate packages are invisible to this tool. A package that exists, is old, and has millions of downloads can still ship malware in a compromised release. depverify's existence and reputation checks say nothing about supply-chain compromise of an otherwise-legitimate package.
• depverify/top_packages.json was hand-vendored, not fetched live, because the environment this project was built in could not reach raw.githubusercontent.com, api.github.com, or any CDN mirror (only pypi.org / files.pythonhosted.org / bare github.com were reachable). It's a ~450-name curated list of well-known packages written from training knowledge, not the real top-5000 hugovk/top-pypi-packages dataset. Run scripts/fetch_top_packages.py from a network with GitHub raw-content access to regenerate the real list before relying on near-name detection for anything beyond obvious cases.
• Attribute-chain verification (--symbols) only covers simple, statically-resolvable chains rooted directly in an imported name (e.g. pandas.DataFrame.flatten). It does not do instance attribute inference, signature/arity checking, type inference, .pyi stub parsing, or follow chains past a function call (requests.get(url).json is not checked past requests.get) or past a reassigned import. See depverify/symbols.py's module docstring for the full scope, and Security note: --symbols and local code execution before enabling it -- unlike the rest of depverify, it imports locally installed code.

Flask HTTP integration

integrations/cortexfeed/flask_blueprint.py is a small, source-only Flask Blueprint exposing verify_text over HTTP (POST /verify). It is not part of the installed depverify package -- pip install depverify[flask] only pulls in the flask dependency it needs, not this file itself. That's intentional: this blueprint is meant to be copied into your own Flask app (it imports only from depverify, nothing from cortexfeed, so it has no hidden dependency on the project it's named after), not imported from a pip-installed depverify. To use it, copy integrations/cortexfeed/flask_blueprint.py from a source checkout into your own project and register the blueprint it defines:

from flask_blueprint import depverify_bp   # after copying the file in
app.register_blueprint(depverify_bp)

See the file's own module docstring for the request/response shape.

Project layout

depverify/
├── depverify/           # the library + CLI -- this is what `pip install depverify` ships
├── integrations/cortexfeed/   # source-only Flask blueprint; copy into your app, not pip-installed
├── tests/                # unit tests (mocked, zero network) + eval/ (live network)
└── scripts/               # fetch_top_packages.py: regenerate the vendored top-package list

Development

pip install -e ".[flask,dev]"
pytest tests/ --ignore=tests/eval     # unit tests, zero network
python tests/eval/run_eval.py          # eval, live network against PyPI/pypistats