Skip to main content

VIPI — Visual Prompt Injection Scanner for Images

Animated VIPI logo reveal

Scan images for prompt injections before an AI agent reads them.
A Python CLI and GitHub Action. Runs fully local or with hosted providers.

Python 3.12 PyPI version OS Support: macOS | Linux | Windows

Images can carry harmful instructions that people overlook but AI models read. VIPI finds the text in an image, including text that standard OCR misses, and checks it for prompt injections. It scans single images, folders and committed image changes.

  • CI/CD and AI code review: scan pull-request images before AI triage tools or coding assistants read them.
  • Image ingestion: scan uploaded or collected images before they enter an AI pipeline.
  • Agents with image input: check screenshots before an agent decides or acts on them.

Quick start

Requires Python 3.12 and uv.

Local

Images never leave your machine. Be aware that local mode downloads several gigabytes of models during vipi setup and has hardware requirements as stated in Models and hardware.

uv tool install --python 3.12 'vipi-scanner[ocr,detector-local]'
vipi setup --ocr-mode balanced
vipi scan image.png --ocr-mode balanced --output report.json

Choose a profile; use the same one for setup and scan:

vipi setup --ocr-mode {quality|balanced|speed}
vipi scan image.png --ocr-mode {quality|balanced|speed} --output report.json

Cloud

Hosted OCR pipeline and injection detection. The defaults are Google OCR and OpenAI's decision API for injection detection. Set only the API keys for the providers you use (keys, .env example).

uv tool install --python 3.12 'vipi-scanner[ocr]'
vipi setup --modus cloud
vipi scan image.png --modus cloud --output report.json

Choose providers per scan:

vipi scan image.png --modus cloud \
  --ocr-provider {google|openai|mistral} \
  --decision-provider {openai|jev} \
  --output report.json

If vipi is not on PATH, run uv tool update-shell and restart the terminal.

How it works (Overview)

Every scan runs up to three steps. OCR reads the text in the image. If the result is unclear, a visual model compares it with the image; when text is missing, VIPI reads the image again rotated, with more contrast or upscaled. A detector then checks the final text for prompt injections and reports a finding at a score of 80 or more. The mode decides who provides the models: local runs them on your machine, cloud calls the providers' APIs.

VIPI reads the text of an image with OCR, has a visual model review unclear OCR and re-reads the image if text is missing, then checks the final text for prompt injections. The scan ends clean (exit 0), with an injection (exit 1) or incomplete (exit 2). Below each step are its local and cloud models.

Models and memory per mode
Step Local quality (default) Local balanced Local speed Cloud
1 · Read text PaddlePaddle PP-OCRv6 Medium
or Small · Tiny
PaddlePaddle PP-OCRv6 Small
or Medium · Tiny
PaddlePaddle PP-OCRv6 Tiny
or Medium · Small
Google Cloud Vision
or OpenAI · Mistral
2 · Verify OCR visually Cloudflare Clef-Flash 9B
or Liquid AI d1 3B
Liquid AI d1 3B
or Cloudflare Clef-Flash 9B
Liquid AI d1 3B
or Cloudflare Clef-Flash 9B
OpenAI Decisions API
or TypeSafe Jev
3 · Detect injections Cloudflare Clef-Flash 9B
or Liquid AI d1 3B · Meta Prompt Guard 2
Liquid AI d1 3B
or Cloudflare Clef-Flash 9B · Meta Prompt Guard 2
Liquid AI d1 3B
or Cloudflare Clef-Flash 9B · Meta Prompt Guard 2
OpenAI Decisions API
or TypeSafe Jev
Memory, Apple Silicon or GPU ¹ 48 GB unified memory (measured)
or 24 GB VRAM
~16 GB unified memory
or 16 GB VRAM
16 GB unified memory (measured)
or 16 GB VRAM
~8 GB RAM
no GPU needed
Memory, CPU only ¹ ~48 GB RAM ~24 GB RAM ~24 GB RAM ~8 GB RAM

¹ Estimates with the defaults; measured were speed on an Apple M2 with 16 GB and quality on an Apple M4 Pro with 48 GB. Decision models load unquantized: 16-bit on Apple Silicon and CUDA, 32-bit on CPU. Clef-Flash 9B needs ~19 GB in 16-bit and ~38 GB in 32-bit. Disk space, model IDs and pinned revisions are in Models and hardware.

Performance

How well VIPI currently reads text in images and detects prompt injections, measured on the publicly available VIPI evaluation dataset. New images are welcome: they help us test VIPI more thoroughly, and other projects can use the dataset to evaluate their own OCR or injection detection. See contributing to the dataset.

Bar charts of detection rate, false positive rate and words recognized: cloud configurations detect 74 to 85 percent of injections with 0.4 to 4.5 percent false positives and recognize 85 to 88 percent of words; local configurations detect 15 to 19 percent with 0 to 2.2 percent false positives and recognize 70 to 77 percent of words. Exact values and models are in the expandable tables below.

Exact values and models
Cloud OCR Detector Detection rate False positive rate Words recognized
Mistral OCR 4.1 TypeSafe Jev 1.14 81.5% 0.4% 84.6%
Mistral OCR 4.1 OpenAI GPT-6 Luna (Decisions API) 85.0% 4.5% 84.6%
OpenAI GPT-6 Luna OpenAI GPT-6 Luna (Decisions API) 83.3% 4.5% 84.8%
OpenAI GPT-6 Luna TypeSafe Jev 1.14 78.9% 1.3% 86.3%
Google Cloud Vision OpenAI GPT-6 Luna (Decisions API, default) 78.4% 3.6% 88.1%
Google Cloud Vision TypeSafe Jev 1.14 74.0% 0.9% 88.1%
Local OCR Detector Detection rate False positive rate Words recognized
PP-OCRv6 Medium (quality) Clef-Flash 9B (default) 16.7% 0% 77.0%
PP-OCRv6 Medium (quality) Prompt Guard 2 86M 19.4% 1.8% 77.0%
PP-OCRv6 Small (balanced) d1 3B 15.4% 0.4% 72.3%
PP-OCRv6 Small (balanced) Prompt Guard 2 86M 16.7% 1.8% 72.3%
PP-OCRv6 Tiny (speed) d1 3B 15.0% 0.4% 70.0%
PP-OCRv6 Tiny (speed) Prompt Guard 2 86M 18.5% 2.2% 70.0%

Usage

vipi scan label.png screenshot.jpg                       # single images
vipi scan ./images --verbose --output report.json        # folder, print OCR text, save JSON report
vipi scan ./images --exclude 'generated/' --exclude '*.webp'
vipi scan . --diff origin/main --output report.json      # only images changed since origin/main
vipi scan image.png --threshold 90                       # stricter finding threshold

# other detector: prepare it first, scans never download models
vipi setup --ocr-mode balanced --detector prompt-guard
vipi scan ./images --ocr-mode balanced --detector prompt-guard
vipi scan image.png --modus cloud --detector jev

For permanent exclusions, copy .vipiignore.example to .vipiignore.

Format Extensions Notes
PNG .png Static images only; animated PNGs are rejected
JPEG .jpg, .jpeg EXIF orientation is applied
WebP .webp Static images only; animated WebPs are rejected

A score at or above the threshold (default 80) is a finding. Size limits, skipped files and what a score does not tell you are in Limitations.

Every option, credential and report field is in the CLI reference.

GitHub Actions

Scan the images a pull request adds or changes. This example uses cloud mode with the default providers, Google Cloud Vision for OCR and OpenAI for decisions:

name: VIPI image scan
on:
  pull_request:
    branches: [main]
permissions:
  contents: read
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          fetch-depth: 0
      - uses: EnXan/VIPI@v0.9.0
        with:
          modus: cloud
        env:
          GOOGLE_CLOUD_API_KEY: ${{ secrets.GOOGLE_CLOUD_API_KEY }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
        if: always()
        with:
          name: vipi-report
          path: vipi-report.json

Note: Local mode is not recommended on standard GitHub-hosted runners. They have no GPU and too little memory for the local models, so scans can take very long or fail. In our test on a standard runner, balanced took about 9 minutes for two images, plus 2 minutes of setup; quality could not even load its model. Self-hosted or larger runners with enough memory, ideally with a GPU, can run local mode, but it remains slower than cloud mode.

Other providers are set with ocr-provider and decision-provider and need their own keys; see Action inputs. Fork PRs normally get no secrets, so their scans cannot reach the providers.

Workbench

The workbench is a visual companion to the CLI that makes it easier to inspect scan results, compare evaluations and edit the dataset in your browser. It runs on your machine and uses the same scan pipeline as vipi scan, with local or cloud settings.

  • Scan an image: choose, drop or paste an image. Inspect the verdict, injection probability, recognized text and individual OCR passes, along with scan time and API cost.
  • Evaluate: run a configuration on the dataset's development split, compare OCR and detection results with a saved baseline, and download reports. Runs can be stopped and resumed.
  • Dataset: add images, edit labels and reference text, and submit your changes as a pull request on Hugging Face.

The workbench is included in the repository, not the PyPI package. With Python 3.12 and uv installed, run:

git clone https://github.com/EnXan/VIPI.git
cd VIPI
uv sync --extra ocr --extra detector-local --extra research --extra test
uv run vipi setup
uv run python -m research.workbench serve

Open localhost:8000. The default is local Quality; vipi setup downloads its models, so check the hardware requirements first. For cloud scans, skip local model setup, configure the provider API keys and choose Cloud under Change settings. Cloud scans incur API charges.

For Evaluate and Dataset, download the evaluation dataset from the repository root before starting the workbench:

uv run python -m research.download

See the workbench guide for settings, baseline comparisons and dataset contributions.

Documentation

License

VIPI's code, documentation and assets use the Apache License 2.0. Models, dependencies, datasets and provider marks keep their own terms.

Metadata

Release files for vipi-scanner 0.9.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for vipi-scanner 0.9.0
File Size Uploaded Provenance Actions
vipi_scanner-0.9.0.tar.gz 189.2 kB Without provenance Details for vipi_scanner-0.9.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for vipi-scanner 0.9.0
File Interpreter ABI Platform Actions
vipi_scanner-0.9.0-py3-none-any.whl Python 3 none any Details for vipi_scanner-0.9.0-py3-none-any.whl

Total release size: 299.3 kB

Release files / vipi_scanner-0.9.0.tar.gz

Download URL vipi_scanner-0.9.0.tar.gz
Size 189.2 kB
Tags Source
SHA-256 checksum
How to use checksums
a5dfec6cd099568debce7095087b12a783cc1c0620596783b2cd6b86246cc86a
BLAKE2b-256 checksum
How to use checksums
cbbb16d986efd867a468a77c5ba18d31953553170fd4959daab786cf753aac04
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / vipi_scanner-0.9.0-py3-none-any.whl

Download URL vipi_scanner-0.9.0-py3-none-any.whl
Size 110.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8310204a7f03c5fea4e3ca4c76130e06b70ffbf657c22d76dcf5bf54957872cb
BLAKE2b-256 checksum
How to use checksums
91124979973add91e7c21ae03d7b1b431835c322e5a41e17a4f4d45ba80b8079
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for vipi-scanner 0.9.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page